DEV Community: Coach Beard Jr

Python learning on Steroids

Coach Beard Jr — Mon, 25 Mar 2024 10:46:03 +0000

Recently i am going through Network and Security in Python book and wanted to write and test the scripts myself.

Along the way I wanted to learn and code using Vim editor.

So my target is this

Continue learning Vim - there is a great tutorial by fireship (search for "fireship vim tutorial") in youtube I am not giving any details/tricks about Vim, that video explain better than me.
I want to keep all my notes at a single place. In a single file is much better.
Along the way I also want snippets of code in between my notes lets call it NOTES.MD

At the bottom of the page you can find the link to github repository where this is used in practice.

All the things started well, until one moment when i coded my first basic script and then wanted to code the second script and run it.

Essentially I need to create a new file to execute the second piece of code but need to keep my notes in a single file and also need to attach code snippets in NOTES.MD

Its just irritating to create multiple files and also to copy paste the same code snippet in NOTES.MD also its just repetition work, which is one thing i hate to do the most as i am lazy.

Putting on thinking hats

I found a way to solve this problem

The notes -> write them in the comments

All the scripts write them in a function and call the respective function to execute the respective method/function/script

An example will be like this

import os

'''
    os.walk(path) 
navigates all the directories in the provided path directory, and
returns three values: _the path directory_, _the names of sub-directories_, _list of filenames in
the current directory path
'''

def os_walk():
    file_count = 0
    for root, directories, files in os.walk(".", topdown=False):
        file_count += len(files)
        for file in files:
            print(f"[+] {os.path.join(root, file)}")
            for sub_folder in directories:
                print(f"[+][+] {sub_folder}")
    print(f"[=] Total files in the current directory: {file_count}")

if __name__ == '__main__':
    os_walk()

Now from the above code snippet, I have my notes about something also the code snippet which i can call from the main method.

If the function needs some arguments then we provide them from the main function as well.

All is well and good to go.

BUT WAIT
We can still enhance this right??

Thinking about this, the problems we face every time we run the new function are

We need to modify the main function logic to call the necessary function
Also we have to keep in mind that we might not be knowing before hand the parameters that a function needs. (in my case this is one of the issue, since we are the ones defining the function we might not face this issue, but i want to solve this as well)

Points to note

User will provide which function to execute
User will provide necessary arguments/parameters via command line arguments

So the command should be

python main.py -f some_random_function --args "{'args1':'values'}"

This command should execute the some_random_function with arguments as 'args1' which value is 'values'

To capture the command line arguments lets use optparse

from optparse import OptionParser

def capture_arguments():
    parser = OptionParser()
    parser.add_option("-f", dest="function", help="Function to execute")
    parser.add_option("--args", dest="args", help="Arguments needed for functions to execute", default={})
    return parser.parse_args()

Now we have to modify the main method to call the respective function that user mentions via command line arguments

Now to call the main method from the function we need to use the globals function that is provided by python.

Python globals() function is a built-in function in Python that returns the dictionary of the current global symbol table.

A symbol table is a data structure that contains all necessary information about the program such as variables, methods, classes etc...

If we want to call a function say sum we can use globals function to call the sum function as globals()['sum']()

In this manner, we are passing which function to execute via command line arguments which we are capturing via optparse library, which returns the arguments and its parameters.

defined below is the code snippet for the main function which call the function and executes the function that we provide via command line arguments.

if __name__ == '__main__':
    (options, args) = capture_arguments()
    func = options.function
    print(f"[=] Executing {func} function")
    globals()[func]()

This is all working good.

What if we need some parameters. Sure we can call input() function whenever these values are required within the function call itself.

But I want to pass parameters from cli itself. (Another enhancement)

Lets begin another journey.

Since we dont know the amount of arguments and the positional arguments that are required by the function we send "kwargs" to the function as parameters.

def random_function(**kwargs):
    #some code
    arg = kwargs.get('arg', None)

if __name__ == '__main__':
    (options, args) = capture_arguments()
    func = options.function
    args = options.args
    print(f"[=] Executing {func} function")
    globals()[func](args)

But the above snippet wont work. The function is expecting dictionary of positional arguments but we giving a string input which then throws TypeError error

To resolve this, lets convert string to dictionary using eval() function

args = eval(options.args)

This is done, but still we are facing issues that if we run this code we will get error stating that random_function() is not exepecting any positional arguments.

For this we convert the args again as

args = **eval(options.args)

This is similar to

def func(arg1=value1, arg2=value2):
    #--- snip----

args = {'arg1':'value1', 'arg2':'value2'}

def func(**args):
    #--- snip----

The above code snippets represents they way we can pass the keyword arguments as a dictionary.

so we can modify our main method code as

if __name__ == '__main__':
    (options, args) = capture_arguments()
    func = options.function
    print(f"[=] Executing {func} function")
    args = **eval(options.args)
    globals()[func](args)

Now still there is one more issue pending,
Not all the functions require arguments to execute the code within it.

So we include try except block in our main method
The function that receive unexpected arguments results in TypeError error

if __name__ == '__main__':
    (options, args) = capture_arguments()
    func = options.function
    print(f"[=] Executing '{func}' function")
    try:
        print("[=] Trying to execute function with passing arguments")
        globals()[func](**eval(options.args))
    except TypeError as error:
        print("[=] Executing function without passing arguments")
        globals()[func]()

Hence the problem resolved.

Now you can create multiple functions in the same file and pass in the respective method name and arguments to execute the specific section of code.

We can extend this to multiple files as well

Create a sample files called sample.py and sample2.py
And write some dummy functions in both the files. Say def1, def2 in sample.py file and def3, def4 in sample2.py

Now create a main.py file and add the below contents in the file

from sample import *
from sample2 import *

from optparse import OptionParser

def capture_arguments():
    parser = OptionParser()
    parser.add_option("-f", dest="function", help="Function to execute")
    parser.add_option("--args", dest="args", help="Arguments needed for functions to execute", default={})
    return parser.parse_args()

if __name__ == '__main__':
    (options, args) = capture_arguments()
    func = options.function
    print(f"[=] Executing '{func}' function")
    try:
        print("[=] Trying to execute function with passing arguments")
        globals()[func](**eval(options.args))
    except TypeError as error:
        print("[=] Executing function without passing arguments")
        globals()[func]()

Now to execute function def1 with arguments val=[1,2], out=False we can run the command

python main.py -f def1 --args "{'val':[1,2],'out':False}"

There you go.

Some python steroids drugs which you might not need at all in your lifetime. :)

Link to code -> Python_On_Steroids

Automate Windows Theme Changes Based on Time with a Batch Script

Coach Beard Jr — Mon, 13 Nov 2023 05:54:21 +0000

Lets try to automate the auto-theme change in windows using dll's

Why you ask?

Well nothing much, I just need an excuse to learn and know how and where dll's can be used in windows
So starting we do the most basic stuff which is solve our daily issues.
One of my such issues is I want light windows theme during day time and dark theme in night time, so I am trying to automate this process.

Lets get started, shall we...

@echo off
setlocal enabledelayedexpansion

@echo off: This command disables command echoing, which prevents the console from displaying the actual commands. To make the output of batch scripts cleaner, it is frequently used at the start of the script.
setlocal enabledelayedexpansion: This command enables delayed variable expansion. It's necessary when using variables inside a block of code (inside parentheses), as it allows you to access the updated values of variables within the block. In this script, it's used within the for loop to update the current_hour variable.

rem Get the current time in hours (24-hour format)
for /f "tokens=1,2 delims=:" %%a in ("%time%") do (
    set /a "current_hour=%%a"
)

rem: This is a comment or remark. It's used to insert comments into the batch script that describe what a certain piece of code accomplishes. It serves as a sign that we are receiving the current time in this instance.
for /f "tokens=1,2 delims=:" %%a in ("%time%") do ...: This for loop is used to split the current time (in the format HH:MM:SS.ss) and extract the hour (HH) part. It uses : as the delimiter, and the tokens=1,2 option specifies which parts of the string to capture.
set /a "current_hour=%%a": This line assigns the extracted hour to the current_hour variable. The /a option tells the set command to treat the right-hand side as an arithmetic expression.

rem Set the time ranges for theme changes
set morning_start=6
set morning_end=11
set afternoon_start=12
set afternoon_end=17
set evening_start=18
set evening_end=23

Here, we define time ranges for when different themes should be applied. The script considers the morning (6 AM - 11 AM), afternoon (12 PM - 5 PM), and evening (6 PM - 11 PM) periods. These time ranges are stored in variables like morning_start, morning_end, etc.

rem Choose the theme based on the current time
if !current_hour! geq %morning_start% if !current_hour! leq %morning_end% (
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Themes\Personalize" /v AppsUseLightTheme /t REG_DWORD /d 1 /f
) else if !current_hour! geq %afternoon_start% if !current_hour! leq %afternoon_end% (
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Themes\Personalize" /v AppsUseLightTheme /t REG_DWORD /d 1 /f
) else if !current_hour! geq %evening_start% if !current_hour! leq %evening_end% (
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Themes\Personalize" /v AppsUseLightTheme /t REG_DWORD /d 0 /f
)

Here, the script checks the current_hour against the defined time ranges for the different parts of the day. If the current hour falls within a specified range, it uses the reg command to change the Windows theme settings.
The reg add command modifies the Windows Registry. In this example, it sets the "AppsUseLightTheme" value under the "Personalize" key to 1 (for light theme) during the morning and afternoon, and 0 (for dark theme) in the evening. The /t REG_DWORD option specifies the data type as a DWORD.

rem Trigger a theme change
rundll32.exe %SystemRoot%\System32\shell32.dll,Control_RunDLL %SystemRoot%\System32\desk.cpl desk,@Themes /Action:OpenTheme /file:"Path\To\Your\Theme"

Finally, the script uses rundll32.exe to open the Windows theme control panel. This line of code triggers a theme change based on the specified theme file path. Be sure to replace "Path\To\Your\Theme" with the actual path to the theme file you want to apply.

This batch script, which can be manually run or scheduled to run at predetermined periods, automates the process of changing the Windows theme dependent on the time of day.

Digital Certificates in Communication - What are THOSE???

Coach Beard Jr — Sun, 12 Nov 2023 14:03:55 +0000

Contents Covered

Public Key Infrastructure ( #PKI )
Encryption, Hashing, Digital Signatures
Installing and configuring PKI solutions
Secure Sockets Layer ( #SSL )
Transport Layer Security ( #TLS )

Prerequisite Knowledge

TCP/IP - how it works and what is it.
Web server/browser configuration - have a general sense of how you might go about configuring a web browser/server

SSL - Secure Socket Layer
TLS - Transport Layer Security

Public Key Infrastructure

Understanding Cryptography

SSL/TLS are security protocols that are used to secure network communications

Cryptography is a method of securing data such that it is trusted and is viewable only by authorized parties

For this to work, cryptographic keys need to be stored somewhere in order to partake in securing things like network communications

Cryptographic Key Storage ways
-> PKI certificate
-> Smart Card/Common Access Card(CAC)
-> File
-> Trusted Platform Module(TPM)
-> Token Device

General Encryption Process

Plaintext is fed into an encryption algorithm
A key is also used with the encryption algorithm
The encryption algorithm results in encrypted data (ciphertext)
Only those parties with a decryption key can decrypt the ciphertext

Symmetric Cryptography

Uses a single secret key
The secret key encrypts and decrypts the data
All parties that either trying to encrypt or decrypt needs this key
This key must be kept secure

Symmetric Encryption Algorithms

Algorithm	Max. Key size(bits)
AES	256
RC4	2048
3DES	168
Blowfish	448

Asymmetric Cryptography

Uses two mathematically related keys
Public key and private key are used in implementation of these type of algorithms

Asymmetric Encryption Algorithms

Algorithm	Max. Key size (Bits)	Description
RSA	4096	-
Diffie-Hellman	n/a	This is more of a key exchange mechanism implemented algorithm
ElGamal	2048	-
ECC	256	-

Asymmetric Email Encryption Process

Sender encrypts the plaintext with receivers public key
Receiver decrypts the encrypted text received using his private key.

SSL/TLS Network Security process

Client sends the list of supported ciphers to web server
Server sends cipher that will be used along with the PKI certificate, which includes the servers public key and the servers hostname
Client generates and sends a unique session key(symmetric key)

Now to secure that symmetric key which is session key, it is encrypted with web servers public key. The web server then decrypts it with its private key

PKI hierarchy

PKI is a collection or hierarchy of digital security certificates.

These certificates always contain public keys of the registered entities
These certificates can also contain private keys which is are defined by the entities in some specific cases for authentication or authorization

One of such components of PKI hierarchy is

-> Certificate Authority (CA)
This component is used to issue new certificates to entities
It can also renew, revoke certificates
This CA also maintains the Certification Revocation List(CRL) which is essentially a list of serial number of revoked certificates

-> Registration Authority (RA)
Also called subordinate CA. Also used to manage certificates

-> Certificate Revocation List (CRL) or Online Certificate Status Protocol (OCSP)
verification of certificate validity using a serial number

-> Certificate Template
Blueprint used when issuing certificates
This can be customized depending on your needs and how the resultant issued PKI certificates are to be used

-> Certificate
contains subject name, signature of CA, expiry information, public/private key

Revoked certificates cannot partake in part of secure communications

In case of multi-tier PKI hierarchy, from a security perspective, it makes sense to keep the certificate authority at the top of the hierarchy offline until its needed, since we have registration authority doing the work on behalf of the top certificate authority

Certificate Authorities

Certificate/registration authorities have a longer validity period than the issued certificates

Can also publish certification revocation list over a variety of protocols including things like HTTP and this certificate revocation list can then be retrieved by clients before they partake in communication that would PKI certificates

Chain of Trust
The digital signatures of the certificate authority will exist in all of the certificates that it issues
-> We trust CA that implies we trust all the certificates that it issue

SSL/TLS are directly related to PKI certificates

PKI certificates are also called X.509 certificates as it follows X.509 standard

What type of data is stored in PKI certificates?

Version number
1. defines the X.509 or PKI version number
Serial number
1. used to track certificates such as through certification revocation list,
CA digital signature and Algorithm used
validity period
Certificate usage details
Subject name such as URL or email address or domain name etc..
Public or Private keys

We can use wildcard certificates for multiple DNS second level domain names via the subject
This means that the certificate is issued to top-level DNS domain and all the subdomains are automatically trusted using single PKI certificate

OCSP Stapling
certificate owner checks the CA for its status periodically, such as a web server with an SSL/TLS certificate
Clients connecting to the secured website receive OCSP status for that website
In this case, clients do not have to query the OCSP responder for website certificate status separately

PKP - Public Key Pinning
This is information that is sent between the client web browser and web server so that trusting devices can download a trusted copy of a server certificate which includes public key
In this we store a copy of that server certificate public key locally on the client device
Upon future connections to the server, clients require that the server provide a fingerprint that has been previously pinned
By this we are mitigating certificates issued to known hosts from unauthorized CAs such as <mail.google.com> being issued a certificate from an attackers self-signed CA
These also have a certified lifetime

Certificate Lifecycle

It starts with certificate request, this can be manual or it can be automated
If everything is okay, then the certificate is issued, then it can be used according to its purpose
If the certificate is expired then new certificate must be created and digitally signed and to be send to receiver

Certificate request
1. depending on the algorithm being used and software is used, public and private key of the entity is created and then public keys are made available to trusted third parties
2. Certificate Signing Request(CSR) is generated next, which includes a unique public key commonly in PKCS #10 format
3. This CSR is sent to certificate authority for signing
Certificate issuance
1. require admin approval before certificate is issued
2. certificate is stored in the device trusted certificate store or other media such as a smart card
Certificate usage
1. apps can first verify the validity of a certificate before using them (via CRL or OCSP)
Certificate revocation
Certificate renewal
Certificate expiry

SSL vs TSL

Both uses PKI certificates and related keys to secure network communication

Both in conjunction with PKI certificates, allow for data encryption and confidentiality
Also allows for digital signatures and hashing for authentication, integrity and non-repudiation

Both protocols are application specific, it must be configured separately for HTTP and SMTP and similar protocols

Security protocol downgrade attacks are common and are problems with the current standard of SSL protocols and hence it is recommended to use TLS instead of SSL unless its necessary

For security reasons, disable SSL and use TLS >v1.1

Hashing and Digital Signatures

Hashing doesn't provide data confidentiality

Used to verify the integrity of network messages, files and machine boot-up settings

Used in both SSL and TLS
Uses a one-way algorithm that results in a unique value from which we cant generate input value except using the brute-force method

Digital Signatures provides data authentication, integrity and non-repudiation
This is done by using the private key of the user, so if a message is sent by a user that user sings the message using his private key so that once we decrypt it with the users public key we can actually verify the user authentication for the message sent.

What are the different types of DNS records, and what are their specific purposes?

Coach Beard Jr — Sat, 04 Nov 2023 10:28:19 +0000

DNS (Domain Name System) records are used to manage and translate domain names into IP addresses, allowing computers to locate and communicate with each other over the internet. There are various types of DNS records, each serving a specific purpose. Here are some common types of DNS records and their purposes:

A (Address) Record:
- Purpose: Maps a domain name to an IPv4 address.
- Example: example.com IN A 192.168.1.1
AAAA (IPv6 Address) Record:
- Purpose: Maps a domain name to an IPv6 address.
- Example: example.com IN AAAA 2001:0db8:85a3:0000:0000:8a2e:0370:7334
CNAME (Canonical Name) Record:
- Purpose: Creates an alias for an existing A or AAAA record. It points one domain name to another.
- Example: www.example.com IN CNAME example.com
MX (Mail Exchanger) Record:
- Purpose: Specifies the mail server responsible for receiving email messages for a domain.
- Example: example.com IN MX 10 mail.example.com
TXT (Text) Record:
- Purpose: Used for storing text-based information, such as SPF (Sender Policy Framework) records, DKIM (DomainKeys Identified Mail) keys, and other human-readable data.
- Example: example.com IN TXT "v=spf1 include:_spf.example.com ~all"
SRV (Service) Record:
- Purpose: Specifies the location of services, such as SIP, XMPP, and other network services.
- Example: _sip._udp.example.com IN SRV 0 5 5060 sipserver.example.com
NS (Name Server) Record:
- Purpose: Specifies the authoritative name servers for a domain.
- Example: example.com IN NS ns1.example-dns.com
PTR (Pointer) Record:
- Purpose: Used for reverse DNS lookups to map an IP address to a domain name.
- Example: 1.1.168.192.in-addr.arpa IN PTR example.com
SOA (Start of Authority) Record:
- Purpose: Contains administrative information about a zone, including the primary name server and contact information.
- Example: example.com IN SOA ns1.example-dns.com admin.example.com 2023102401 7200 3600 604800 3600
SPF (Sender Policy Framework) Record:
- Purpose: Specifies which IP addresses are allowed to send email on behalf of a domain, helping to prevent email spoofing and phishing.
- Example: example.com IN TXT "v=spf1 include:_spf.example.com ~all"

These DNS record types serve different functions and are essential for the proper functioning of the internet. They allow domain names to be associated with various services and resources, such as web servers, email servers, and more. The choice of which record type to use depends on the specific requirements of the domain and the services it provides.

Auto Publish/Release - GitHub Actions

Coach Beard Jr — Sun, 29 Oct 2023 15:03:28 +0000

Continuous Integration(CI) is often coupled with the idea of Continuous Delivery(CD).

For releasing or publishing our code automatically, we will use the action Release Drafter

Now we have to configure our actions file.

We can create new workflow file or we can add this workflow to our already present files.

If you want to add the release drafter workflow to already present action file, then the snippet is

name: <PreviousContent>

on:
    <PreviousContent>

jobs:
    allPreviousJos:
        <PreviousJobsContent>

    update_draft_release:
        runs-on: ubuntu-latest
        needs: [ All previous jobs comma separated ]
        steps:
            - uses: toolmantim/release-drafter@v5.2.0
              env:
                  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

If you see the above code snippet we added one more keyword needs

needs

This just means that this particular jobs requires the mentioned jobs to be executed first then run this particular job.

In our code, we need our publish job to be executed once all our tests and build process is executed successfully, so in that sense we want to execute the build, test, and all other steps to run perfectly and then execute this publish method.

jobs:
    hi-mom:
        runs-on: ubuntu-latest
        steps:
            run: echo Hi MOM
    hi-dad:
        runs-on: ubuntu-latest
        steps:
            run echo Hi DAD

    hi-bro:
        runs-on: ubuntu-latest
        needs:
            - job1
            - job2
        steps:
            run: echo "Hello BRO"

If you are adding new workflow file for the auto-publish process

create new file with whatever name you want and have the below content in that file

name: Release Drafter

on:
  push:
    # branches to consider in the event; optional, defaults to all
    branches:
      - master
  # pull_request event is required only for autolabeler
  pull_request:
    # Only following types are handled by the action, but one can default to all as well
    types: [opened, reopened, synchronize]
  # pull_request_target event is required for autolabeler to support PRs from forks
  # pull_request_target:
  #   types: [opened, reopened, synchronize]

permissions:
  contents: read

jobs:
  update_release_draft:
    # permissions:
      # write permission is required to create a github release
      # contents: write
      # write permission is required for autolabeler
      # otherwise, read permission is required at least
      # pull-requests: write
    runs-on: ubuntu-latest
    steps:
      # (Optional) GitHub Enterprise requires GHE_HOST variable set
      #- name: Set GHE_HOST
      #  run: |
      #    echo "GHE_HOST=${GITHUB_SERVER_URL##https:\/\/}" >> $GITHUB_ENV

      # Drafts your next Release notes as Pull Requests are merged into "master"
      - uses: release-drafter/release-drafter@v5
        # (Optional) specify config name to use, relative to .github/. Default: release-drafter.yml
        # with:
        #   config-name: my-config.yml
        #   disable-autolabeler: true
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Once this is done,
The release drafter requires a template to create all the changes and updates that the release had and little like documentation for updated version of release thing.

So create this template in .github folder with below contents

template: |

  ## What's Changed

  $CHANGES

now commit the changes.

Remember that the template should be named as release-drafter.yml in .github folder in your repo

Here comes the good part

Make some random changes in the repo but not in main branch, let those changes be anything

Here i will do some changes regarding in the index.js file and commit them in the testing branch

Once the necessary changes are done, raise the pr and wait for workflows to run and see whether they all checks are passed or not, if not resolve the issue

Releases

Now if you click on releases link in your repo, you will see a draft release docs already being created which is created by github-actions

The draft version may look something like this

In this i have modified some content in the draft release which is title of the release

You can also edit this draft or delete this draft

Now release your package...

GitHub Actions part 3 DONE...

Continuous Integration - GitHub Actions

Coach Beard Jr — Sat, 28 Oct 2023 17:29:20 +0000

Base Repository for this example is Github-Actions-Playground

Topics Covered

NodeJs Workflow changes from generated default workflow
strategy -- what is strategy in github actions
matrix -- what is the use of this in github actions
include -- what is the use of this in github actions
workflow_dispatch -- what is this in github actions
Actual workflow of our nodejs action
Building the test node js app.
Then seeing whether the actions are running correctly or not

The playground repository contains the javascript code and we are going to use NodeJs environment to test and build our code.

For this you can fork this repo or create new one in another language maybe Scala or Python to actually get your hands dirty in all ways.

Go to repository -> Actions Tab -> Search for lang CI action (in my case it is NodeJs env) -> click on Configure button

If you are not using NodeJs workflow you can skip this section

The changes I made from the default yaml file

I don't need Node 14.x version
Added name fields in all npm commands

strategy

This is used to matrix strategy for jobs, (which we discussed in github-actions series in dev.to post) Intro to GITHUB ACTIONS

A matrix strategy lets you use variables in a single job definition to automatically create multiple jobs that are based on the combinations of the variables.

Example case is that you want to test your code in multiple versions and in multiple os systems

For this we use matrix

matrix

Use this to define your variables of different job configurations.
Within your matrix, you can define one or more variables followed by an array of values.

Examples:

name: github matrix example

jobs:
    matrix_example:
        strategy:
            matrix:
                members: ["MOM", "DAD", "BRO"]
        steps:
            - name: Say hello to everyone
              run: echo Hello ${{matrix.members}}

You can also use multiple variables in the matrix

name: github matrix example

jobs:
    matrix_example:
        strategy:
            matrix:
                members: ["MOM", "DAD", "BRO"]
                os: [windows-latest, ubuntu-latest]
        runs-on: ${{matrix.os}}
        steps:
            - name: Say hello to everyone
              run: echo Hello ${{matrix.members}}

This above workflow will run for each possible combination of the variables "os" and "members"

To add new values to the matrix variable you can use include key to do that.

include

For each object in the include list, the key:value pairs in the object will be added to each of the matrix combinations if none of the key:value pairs overwrite any of original matrix values

If the object cannot be added to any of the matrix combinations, a new matrix combination will be created instead.
The original matrix values will not be overwritten, but added matrix values van be overwritten

strategy:
  matrix:
    fruit: [apple, pear]
    animal: [cat, dog]
    include:
      - color: green
      - color: pink
        animal: cat
      - fruit: apple
        shape: circle
      - fruit: banana
      - fruit: banana
        animal: cat

Now the runner have the following combinations to run the job

{fruit: apple, animal: cat, color: pink, shape: circle}
{fruit: apple, animal: dog, shape: circle, color: green}
{fruit: pear, animal: cat, color: pink}
{fruit: pear, animal: dog, color: green}
{fruit: banana, animal: cat}
{fruit: banana}

workflow_dispatch

To run a workflow manually, the workflow must be configured to run on the workflow_dispatch event

To trigger the workflow event, your workflow must be in the default branch.
Based on your configurations, write access might be required to the repository is required to perform jobs

Actual workflow of our nodejs action

Now coming to the CI part of actions.

create and commit that yml/yaml file to github repo

Once you commit your file, if you go to actions tab, you will one action that is automatically started, well there will be multiple if you follow previous article also, but for time being one NodeJs action will be triggered automatically.

Why you might ask..

Well the reason is the below code snippet in the yaml file

on:
    push:
        branches: ['main']
    pull_request:
        branches: ['main']

    workflow_dispatch:

Now you already know why workflow_dispatch is added.

Moving on we also added push and pull_request that means that for every pull request and commits in that particular branch of main the action will start automatically.

So far we have not clearly seen this actions to clear usage I think.
Like we want to use these actions while we are collaborating right.

So let us create a new branch from the repo and we will make some changes and create a pull request to the default main branch.

Now my main branch name is main and i will be creating a branch testing from the main branch and I will add a new test case in this testing branch in index.test.js file

test('that weekNum returns a number', () => {
  expect(weekNum).toBeDefined();
});

Add the above code snippet in index.test.js file.

Now create a pull request(pr) from your testing branch to your main branch.

Once the pr is raised you will see the checks from nodejs action file starts running.
You will see something like this

Now you can see the power of github actions

You don't have to worry about first merging the changes and go through hell to find out that there are some conflicts in production and then try to find out what are the changes that need to be done again.

Instead you will implement all those checks for all the PRs and if these checks are successful you can merge these changes and don't have to worry about fixing the errors in production because you are checking them before releasing them or better even before merging new changes to base code.

So the install and test process of our workflow is done and there are no errors.

Lets go through build process

Build process

The Github Actions workflow allows users to take advantage of vast selection of open source tools and solutions. Since actions workflow runners are in themselves VMs running on cloud servers.

This is different from GitHub app workflow, which requires you to host it on your own server

For this we will use Parcel to build our scripts.

First install the module

npm i -D parcel

Since we have only one file we can build using the command

npx parcel index.js

Else in out package.json file add the following configuration in scripts variable

"build": "parcel build index.js

Once after adding, this should be the outlook of your package.json file

{
  "name": "github-actions-playground",
  "version": "0.1.0",
  "description": "playground area to learn about github-actions",
  "main": "index.js",
  "scripts": {
    "test": "jest",
    "build": "parcel build index.js" //here we are directly pointing to the file we need to build
  },
  "author": "vijay2249",
  "license": "ISC",
  "devDependencies": {
    "jest": "^29.7.0",
    "parcel": "^2.10.1"
  }
}

Now once the changes are done, commit your changes and then the actions will run automatically as you mentioned in yml file that the action should run on each pull request and on each push to global repo.

So now scroll down to the build step and now see for yourself how the output looks like in the build step.

Consider this as homework or getting your hands dirty.
Side note -> check the errors carefully if there are any, for what and where to make changes in your code.

BTW starter-workflows <- this is official github repository with some examples on github actions.
Just go through them and you have to spend less time reading these type of extra peoples notes :)

:) :)

Lets continue the next part some other time.

Servlets & Filters - SpringSecurity

Coach Beard Jr — Fri, 27 Oct 2023 16:23:56 +0000

Servlets

In Java web apps, Servlet container (web server) takes care of translating the HTTP messages for
java code to understand. One of mostly used servlet container is Apache Tomcat.

Servlet Container converts the HTTP messages into ServletRequest and hand over to Servlet method as a parameter

Similarly, ServletResponse returns as an output to Servlet Container from Servlet. So everything we write inside the Java web apps are driven by Servlets

Filters

Filters inside Java Web Applications can be used to intercept each request/response and do some pre-work before our business logic. So using the same filters, Spring Security enforce security based on our configurations inside a web application

Spring Security Internal Flow

Spring Security Filters

A series of spring security filters intercept each request and work together to identify if authentication is required or not.

If authentication is required, accordingly navigate the user to login page or use the existing details stored during the initial authentication

Authentication

Filters like UsernamePasswordAuthenticationFilter will extract username/password from HTTP request and prepare Authentication type object. Because Authentication is the core standard of storing authenticated user details inside Spring Security Framework

AuthenticationProvider

AuthenticationProviders has all the core logic of validating user details for authentication

UserDetailsManager/UserDetailsService

UserDetailsManager/UserDetailsService helps in retrieving, creating, updating, deleting the User details from the DB/storage systems

PasswordEncoder

Service interface that helps in encoding and hashing passwords. Otherwise we may have to live with plain text passwords

SecurityContext

Once the request has been authenticated, the Authentication will usually be stored in a thread-local SecurityContext managed by the SecurityContextHolder.

This helps during the upcoming requests from the same user

Sequence Flow

--> Request --> [<Authentication Filters> {AuthorizationFilter, DefaultLoginPageGeneratingFilter, UsernamePasswordAuthenticationFilter}] --> Extract User Credentials --> [<Authentication>{UsernamePasswordAuthenticationToken}] --> authenticate() --> [<AuthenticationManager>{ProviderManager}] --> authenticate() --> [<AuthenticationProvider>{DaoAuthenticationProvider}] --> loadUserByUsername() -->[<UserDetailsService>{InMemoryUserDetailsManager}] --> UserDetails --> [<AuthenticationProvider>{DaoAuthenticationProvider}] --> Authentication --> [<AuthenticationManager>{ProviderManager}] --> Authentication --> [<Authentication>{UsernamePasswordAuthenticationToken} --> Authentication --> [<Authentication Filters> {AuthorizationFilter, DefaultLoginPageGeneratingFilter, UsernamePasswordAuthenticationFilter}] --> Response

Intro to GITHUB ACTIONS

Coach Beard Jr — Wed, 25 Oct 2023 17:54:01 +0000

Contents covered:

jobs -- what are jobs in github workflow
runs-on -- what is this keyword mean
steps -- what are steps in github actions
Environment Variables -- what are env values, and how to use and reuse them
1. Defining environment variables for a single workflow
2. Defining configuration variables for multiple workflows
3. Writing values into GITHUB_ENV
JOBS_OUTPUT -- using one job output by another job

You can create a workflow file configured to run on specific events

Official Docs -> Configuring a workflow and Workflow syntax for GitHub Actions and GitHub Actions

Assuming you know the basics of actions like what are they I am continuing the explanation

Now the basic thing is that the actions must include jobs keyword and its actions

jobs

A job is defined task made up of steps. Each job is run in a fresh instance of virtual environment.

You can define the dependency rules for how jobs run in a workflow file. Jobs can run at the same time in parallel to be dependent on the status of a previous job and run sequentially

Example case:
Workflow can have two sequential jobs that build and test code, where the test job is dependent on the status of the build job. If the build job fails, then test job will not run

runs-on

Github hosts virtual machine to run these jobs as mentioned above, we can mention which OS to run these jobs on

jobs:
    First-job:
        runs-on: ['windows', 'ubuntu']
    second-job:
        runs-on: 'ubuntu'

All the mentioned jobs execute in the same instance of the virtual environment by default but you can change it, allowing the actions in that job to share information using the file system

steps

A step is a set of tasks performed by a job.
Each step in a job executes in the save environment, allowing the actions in that job to share information using the filesystem if necessary.

These steps can run system commands or actions that user require

jobs:
    start-action:
        name: "Add Steps in workflow"
        steps:

            - name: "Hello MOM"
              run: echo "Hello MOM"

Environment Variables

Now some command or actions might require some env variables that we cannot hardcode these values in our code, they might be some username passwords or api keys etc.

You can set a custom variables in two ways

To define an environment variable for use in a single workflow, you can use the env key in the workflow file.
To define a configuration variable across multiple workflows, you can define it at the organization, repository, or environment level.

Now how can we mention them in our actions file

jobs:
    env-example:
        name: "Env vals example"
        runs-on: 'ubuntu'
        steps:
            - name: "Hello MOM"
              run: "hello $WHO"
              env:
               - WHO: MOM

Now github by default gives access to some environment variables which depends on github user and varies from user to user

jobs:
    github-env:
        name:"Github ENV example"
        runs-on: 'ubuntu'
        steps:
            - name: Hello MOM
              run: "Hello $GITHUB_ACTOR"

The complete list of github env variables that github provides by default are here default-environment-variables

Defining environment variables for a single workflow

To set a custom env variable for a single workflow, you can define it using the env key in the workflow file

The scope of custom variable is set by this method is limited to the element in which it is defined.

You can define variables that are scoped

the entire workflow, by using env at the top level of the workflow
the contents of a job within a workflow, by defining the key value pair inside a job
a specific step within a job, by defining that key value pair inside that particular step

name: This is env variable example defining in workflow
on:
    workflow_dispatch
env:
    BOSS: "MOM" %% this env is avaiable for all the jobs/steps in the workflow %%

jobs:
    greeting_job:
        runs-on: ubuntu
        env:
            Greeting: "Good" %% this env is available to all steps within this job %%
        steps:
            - name: "wishes to MOM"
              run: echo "$Greeting $BOSS, have a great $DAY"
              env:
                  DAY: "Friday" %% this env value is available to only this step %%

The commands in the run steps of a workflow, or referenced actions, are processed by the shell you are using on the runner. The instructions in the other parts of a workflow are processed by Github Actions and are not sent to the runner.

Because runner environment variable interpolation is done after a workflow job is sent to a runner machine, you must use the appropriate syntax for the shell that's used on the runner.

Example:
when you use ubuntu these use bash shell, so you must use the syntax $KEY , and for windows env use PowerShell, so you would use the syntax $env:KEY

Now you cant just name these env variables as you want, there are some default env variables that github gives you to use them and you cant name your env keys to be these names

Example in the above code snipper where we used, GITHUB_ACTOR as env value into run command directly you can't create env value with key as GITHUB_ACTOR

If you try to override the value of one of these default variables, the assignment is ignored by github runners

Defining configuration variables for multiple workflows

You can create configuration variables for use across multiple workflows, and can define them at either the organization, repository or environment level

Example, you can use configuration variables to set default values for parameters passed to build tools at an organization level, but then allow repository owners to override these parameters on case-by-case basis

When you define configuration variables, they are automatically available in the vars context,

If a configuration variable has not been set, the return value of a context referencing the variable will be empty string

name: var usage example workflow

on:
  workflow_dispatch:
env:
  # Setting an environment variable with the value of a configuration variable
  env_var: ${{ vars.ENV_CONTEXT_VAR }}

jobs:
  display-variables:
    name: ${{ vars.JOB_NAME }}
    # You can use configuration variables with the `vars` context for dynamic jobs
    if: ${{ vars.USE_VARIABLES == 'true' }}
    runs-on: ${{ vars.RUNNER }}
    environment: ${{ vars.ENVIRONMENT_STAGE }}
    steps:
    - name: Use variables
      run: |
        echo "repository variable : $REPOSITORY_VAR"
        echo "organization variable : $ORGANIZATION_VAR"
        echo "overridden variable : $OVERRIDE_VAR"
        echo "variable from shell environment : $env_var"
      env:
        REPOSITORY_VAR: ${{ vars.REPOSITORY_VAR }}
        ORGANIZATION_VAR: ${{ vars.ORGANIZATION_VAR }}
        OVERRIDE_VAR: ${{ vars.OVERRIDE_VAR }}

    - name: ${{ vars.HELLO_WORLD_STEP }}
      if: ${{ vars.HELLO_WORLD_ENABLED == 'true' }}
      uses: actions/hello-world-javascript-action@main
      with:
        who-to-greet: ${{ vars.GREET_NAME }}

The if conditionals are processed by Github Actions and only the steps where the check resolves as true are sent to the runner.

Writing values into GITHUB_ENV

If you generate a value in one step of a job, you can use the value in subsequent steps of the same job by assigning the value to an existing or new environment variable and then writing this to the GITHUB_ENV environment file.

echo "{environment_variable_name}={value}" >> "$GITHUB_ENV"

steps:
  - name: Set the value
    id: step_one
    run: |
      echo "action_state=yellow" >> "$GITHUB_ENV"
  - name: Use the value
    id: step_two
    run: |
      printf '%s\n' "$action_state" # This will output 'yellow'

The environment file can be used directly by an action, or from a shell command in the workflow file by using the run keyword.

JOBS_OUTPUT

If you want to pass a value from a step in one job in a workflow to a step in another job in the workflow, you can define the value as a job output
Then you can then reference this job output from a step in another job

jobs:
  job1:
    runs-on: ubuntu-latest
    # Map a step output to a job output
    outputs:
      output1: ${{ steps.step1.outputs.test }}
      output2: ${{ steps.step2.outputs.test }}
    steps:
      - id: step1
        run: echo "test=hello" >> "$GITHUB_OUTPUT"
      - id: step2
        run: echo "test=world" >> "$GITHUB_OUTPUT"
  job2:
    runs-on: ubuntu-latest
    needs: job1
    steps:
      - env:
          OUTPUT1: ${{needs.job1.outputs.output1}}
          OUTPUT2: ${{needs.job1.outputs.output2}}
        run: echo "$OUTPUT1 $OUTPUT2"

These outputs are "UNICODE" string, and can be a maximum of 1MB. The total of all outputs in a workflow can be a max of 50MB

Jobs outputs containing expressions are evaluated on the runner at the end of each job. Outputs containing secrets are redacted on the runner and not sent to the github actions

Lets continue this Discussion on GitHub Actions, with examples whenever necessary in upcoming explanation notes.

Testing with auto push and publishing

Coach Beard Jr — Tue, 24 Oct 2023 17:45:31 +0000

this is paragraph

this is italics
All the above is written in body_markdown propertyalso note that main_image property is added. https://wallpaperaccess.com/full/1264330.jpg this as value to property

GRUB. what is it?

Coach Beard Jr — Tue, 24 Oct 2023 07:31:26 +0000

Intro

A boot loader is one of the most important components of Linux operating systems boot process.
When a device starts up, working memory must be loaded with operating system data. The so-called bootloader, usually referred to as a boot program or bootstrap loader, enables this.

Grand Unified Bootloader(GRUB)

The boot process on Linux is a series of activities that occur from the time you press the power on button on your PC until the login screen appears

Stages of boot process

There are four main stages in the boot process

BIOS
1. mainly responsible for loading the bootloader
2. When the computer starts, it runs a Power On Self Test(POST) to make sure that the core hardware such as memory and hard disk is working properly
3. Afterwards, the BIOS will check the primary hard drives Master Boot Record(MBR), which is a section on your hard drive where the bootloader is located
Bootloader
1. Loads the kernel into RAM with a set of kernel parameters
Kernel
1. The kernel primary function is to initialize devices and memory
2. Afterwards it loads the init process
Init
1. Responsible for starting and stopping essential services on your system GRUB is mainly responsible for providing you with an options menu from which you can select the operating system or environment that you want to boot into

GRUB is not only limited to booting into Linux operating system, you can also use it to boot into other operating systems such as windows

The Role of GRUB

Once you select the operating system to boot into, GRUB will load the selected kernel
GRUB uses kernel parameters to know where the kernel is located and other important parameters to use

initrd 1. used for specifying the initial RAM disk
1. BOOT_IMAGE
  1. the location of the linux kernel image
2. root
  1. specifies the location of root file system, used by the kernel to find init which in turn loads the critical services
3. ro
  1. responsible for mounting the file system in read-only mode
4. quiet
  1. hides some system specific messages as your PC is booting
5. splash
  1. used for displaying the splash screen when your system is booting
6. When you are in the GRUB options menu, you can edit kernel parameters by pressing the “E” key

GRUB 2 gives you a lot of flexibility and power when it comes to configuring your bootloader
The /boot/grub directory contains a file named “grub.cfg” which is the main configuration file for GRUB
However, it is advised not to edit the grub.cfg file directly, instead you should edit the “/etc/default/grub” file
When you make changes to the “/etc/default/grub” file, you should make sure to run the command below, so that your changes are written to the grub.cfg file automatically
- sudo update-grub

Test

Coach Beard Jr — Sun, 08 Oct 2023 07:01:10 +0000

A Test Post