DEV Community: Kishan B

Checking out code using github action in legacy runner

Kishan B — Sun, 18 Aug 2024 07:53:35 +0000

Problem

Performing a simple git checkout using official checkout action fails in a legacy self hosted runner running amazon linux 2 because newer versions of nodejs(>= 20) requires a newer version of GLIBC which is not available in these operating systems.

Solution

Use bash to perform the checkout in the action code i.e

replace the line

- uses: actions/checkout@v3

with

- name: Checkout
  run: |
    git clone --depth 1 -b "${GITHUB_HEAD_REF:-$GITHUB_REF_NAME}" "https://github.com/${GITHUB_REPOSITORY}.git" .

Note

--depth 1 performs a shallow-clone there by increasing your performance as we don't need to pull the entire commit history for every pipeline run.
${GITHUB_HEAD_REF:-$GITHUB_REF_NAME} doing this to ensure we always get the name of the branch which triggered the pipelines. This way of doing it makes it trigger agnostic i.e. both push and pull request trigger will work as intended.
. The dot at the end ensures that the checkout happens in the current directory which is the workspace.

How to use tfenv to install older versions of terraform in m1 mac?

Kishan B — Tue, 12 Sep 2023 04:44:48 +0000

To do that assign "TFENV_ARCH" environment variable with value "amd64" like below.

TFENV_ARCH=amd64 tfenv install 0.12.31

How to get the DDL dump of a redshift database

Kishan B — Thu, 12 Jan 2023 05:38:09 +0000

When I was googling "How to get the DDL dump of a redshift database" I found this blog which proposed a way do it but the con of the approach proposed there is it requires us create new objects in redshift just to extract DDL which was not acceptable for me, so I tried and found a better way which is given below

Approach

The approach below makes use of the following commands from redshift and wraps them in a bash script which makes use of the redshift data api

Steps

Ensure awscli and jq command are installed.
Create a script named dumpddl.sh with the following contents and give it executable permissions using command chmod +x ./dumpddl.sh

#!/usr/bin/env bash

CLUSTER_IDENTIFIER="<<your cluster identifier - required only for managed redshift>>"
WORKGROUP_NAME="<<workgroup name - required only for serverless>>"
DATABASE="<<your database>>"
DATABASE_USER="<<your database user>>"

function query_result() {
  query="$1"
  query_id=$(aws redshift-data execute-statement --with-event --db-user "${DATABASE_USER}" --cluster-identifier "${CLUSTER_IDENTIFIER}" --workgroup-name "${WORKGROUP_NAME}" --database "${DATABASE}" --sql "${query}" | jq -r ".Id")
  sleep 2
  aws redshift-data get-statement-result --id "${query_id}" | jq -r ".Records[][0].stringValue"
  echo ""
}

# Replace the below names of tables, views and external tables with the ones you want to take ddl dump on
query_result "SHOW TABLE person"
query_result "SHOW VIEW org_view"
query_result "SHOW EXTERNAL TABLE \"org_schema\".employee_details"

Run following command to get the ddl dump

./dumpddl.sh > ddl_dump.sql

Pros

No need to install anything inside the redshift cluster.
Works from the local machine even if redshift cluster is within a VPC(through redshift data API).
Works for both managed redshift and serverless redshift.

Cons

Requires us to know the list of tables and views for which we want the ddl dump.

Used TDD approach for a Hello World rust cli app

Kishan B — Sun, 26 Jun 2022 14:10:33 +0000

For the first time ever in my life i wrote hello world application using TDD (Test driven development). Don't get me wrong here, i have practiced TDD before for larger applications but never have i done it for an hello world app.

Problem

Write an CLI application that prints out "Hello TDD world!" in STDOUT.

How did i do it?

I created a project using cargo new hello-world-tdd
I added a dependency assert_cmd which lets us test out outputs of any binary program
Created a file tests/cli.rs and I wrote the following test

use assert_cmd::Command;

#[test]
fn test_cli_app_should_print_hello_tdd_world() {
    let mut cmd = Command::cargo_bin("hello-world-tdd").unwrap();
    cmd.assert().success().stdout("Hello TDD world!\n");
}

Run the test using command cargo test
The above test will fail saying the following, this happens because the cargo new had generated a src/main.rs file which printed out "Hello world!"

running 1 test
test test_cli_app_should_print_hello_tdd_world ... FAILED

failures:

---- test_cli_app_should_print_hello_tdd_world stdout ----
thread 'test_cli_app_should_print_hello_tdd_world' panicked at 'Unexpected stdout, failed diff original var
├── original: Hello TDD world!
├── diff:
│   ---   orig
│   +++   var
│   @@ -1 +1 @@
│   -Hello TDD world!
│   +Hello world!
└── var as str: Hello world!

Now that we have a failing test(RED), the next job is to make it green by modifying the contents of the println statement.
Run cargo test again, The test now passes (Green)

The code for this is available at https://github.com/kishaningithub/hello-world-tdd

Use ZAP to Perform DAST (Dynamic Application Security Testing)

Kishan B — Wed, 16 Feb 2022 09:18:57 +0000

Introduction
- What is Dynamic Application Security Testing (DAST)
- What is ZAP
How to use ZAP
- ZAP Scan for API
- Local Run
  - Example - for API with Swagger
  - For SOAP/GraphQL
- CircleCI Integrate
  - Example - for API with Swagger
  - For SOAP/GraphQL
- ZAP Scan for Application (with UI)
  - Local Run for UI app
  - CircleCI Integration for UI app

Introduction

What is Dynamic Application Security Testing (DAST)

Dynamic application security testing (DAST) is a process of testing an application or software product in an operating state. This kind of testing is helpful for industry-standard compliance and general security protections for evolving projects.

What is ZAP

Zed Attack Proxy (ZAP) is a free, open-source penetration testing tool being maintained under the umbrella of the Open Web Application Security Project (OWASP). ZAP is designed specifically for testing web applications and is both flexible and extensible.

How to use ZAP

ZAP Scan for API

You can use zap-api-scan to perform scans against APIs defined by OpenAPI, SOAP, or GraphQL. If your API is protected with authentication, you will need to prepare a token or API key before running the script.

Local Run

Example - for API with Swagger

The following example shows how to run ZAP locally against an API with:

url: http://192.168.1.10
the swagger doc url: http://192.168.1.10/swagger.json
Authentication header name: Authorization
Authentication header value: api_token_value

Steps:

Put your token or API key used for authentication into a configuration file or environment parameters, the following is a configuration file example.

ZAP_AUTH_HEADER_VALUE=api_token_value
ZAP_AUTH_HEADER=Authorization
ZAP_AUTH_HEADER_SITE=192.168.1.10

Note: the ZAP_AUTH_HEADER_SITE value should exclude the http/https protocol and port, e.g. for the target https://www.this-is-a-target.com:8080, the ZAP_AUTH_HEADER_SITE value should be www.this-is-a-target.com.

Run the following command to pull the zap docker image

docker pull owasp/zap2docker-stable

Run the following command to trigger zap-api-scan for an API with swagger doc (please replace the values in color when you use it for your API):

docker run -it --env-file configuration_file_name --rm -v $PWD:/zap/wrk owasp/zap2docker-stable:latest zap-api-scan.py -t http://192.168.1.10/swagger.json -f openapi -r report.html

After executing the above commands, a report.html should be generated in your current directory, it contains the test results.

For SOAP/GraphQL

To run ZAP Scan against SOAP/GraphQL is very similar to the way to run it against Swagger API, the only difference is you need to change the "-f" option in the step 3 command to the following.

For SOAP:

docker run -it --env-file configuration_file_name --rm -v $PWD:/zap/wrk owasp/zap2docker-stable:latest zap-api-scan.py -t http://192.168.1.10/soap_wsdl_url -f soap -r report.html

For GraphQL:

docker run -it --env-file configuration_file_name --rm -v $PWD:/zap/wrk owasp/zap2docker-stable:latest zap-api-scan.py -t http://192.168.1.10/graphql_url -f graphql -r report.html

CircleCI Integrate

Example - for API with Swagger

The following example shows how to integrate ZAP into CircleCI to scan the API with:

url: http://192.168.1.10
the swagger doc url: http://192.168.1.10/swagger.json
Authentication header name: Authorization
Authentication header value: api_token_value

Steps:

Add the following Environment Variables into your project env in the CircleCI

ZAP_AUTH_HEADER_VALUE=api_token_value
ZAP_AUTH_HEADER=Authorization
ZAP_AUTH_HEADER_SITE=target_url

Add the following content in .circleci/config.yml:

jobs:
  scan:
    docker:
      - image: owasp/zap2docker-stable
    steps:
      - run:
            command: |
              mkdir /zap/wrk
              zap-api-scan.py -f openapi -t https://target-url/swagger.json -r report.html
      - store_artifacts:
          path: /zap/wrk
          destination: zap-report

After executing the above commands, a zap-report/report.html should be generated in the pipeline artifacts, it contains the test results.

For SOAP/GraphQL

To integrate ZAP Scan into CI/CD to scan the SOAP/GraphQL API is very similar to the way to run it against Swagger API, the only difference is you need to change the "-f" option in the step 2 .circleci/config.yml file.

For SOAP:

jobs:
  scan:
    docker:
      - image: owasp/zap2docker-stable
    steps:
      - run:
            command: |
              mkdir /zap/wrk
              zap-api-scan.py -f soap  -t https://target-url/soap_wsdl_url -r report.html
      - store_artifacts:
          path: /zap/wrk
          destination: zap-report

For GraphQL:

jobs:
  scan:
    docker:
      - image: owasp/zap2docker-stable
    steps:
      - run:
            command: |
              mkdir /zap/wrk
              zap-api-scan.py -f graphql -t https://target-url/graphql_url -r report.html
      - store_artifacts:
          path: /zap/wrk
          destination: zap-report

ZAP Scan for Application (with UI)

You can use zap-full-scan to perform a full active scan for a web application. If your application is protected with authentication, you will need to prepare an authorization header or cookie before running the script.

Local Run for UI app

The following example shows how to run ZAP locally against an application with:

url: http://192.168.1.10
Authentication header name: Authorization
Authentication header value: authrozation_token_here

Note: Please change the header name to "Cookie" if your application is authenticated by cookie/session.

Steps:

Put your token or API key used for authentication into a configuration file or environment parameters, the following is a configuration file example.

ZAP_AUTH_HEADER_VALUE=authrozation_token_here
ZAP_AUTH_HEADER=Authorization
ZAP_AUTH_HEADER_SITE=192.168.1.10

Run the following command to pull the zap docker image

docker pull owasp/zap2docker-stable

Run the following command to trigger zap-full-scan (please replace the values in color when you use it for your application):

docker run -it --env-file configuration_file_name --rm -v $PWD:/zap/wrk owasp/zap2docker-stable:latest zap-full-scan.py -t http://192.168.1.10 -r report.html

After executing the above commands, a report.html should be generated in your current directory, it contains the test results.

CircleCI Integration for UI app

The following example shows how to integrate ZAP into CircleCI to scan the application with:

url: http://192.168.1.10
Authentication header name: Authorization
Authentication header value: authrozation_token_here

Steps:

Add the following Environment Variables into your project env in the CircleCI:

ZAP_AUTH_HEADER_VALUE=api_token_value
ZAP_AUTH_HEADER=Authorization
ZAP_AUTH_HEADER_SITE=target_url

Note: Please change the header name to "Cookie" if your application is authenticated by cookie/session.

Add the following content in .circleci/config.yml:

jobs:
  scan:
    docker:
      - image: owasp/zap2docker-stable
    steps:
      - run:
            command: |
              mkdir /zap/wrk
              zap-full-scan.py -t https://target-url -r report.html
      - store_artifacts:
          path: /zap/wrk
          destination: zap-report

After executing the above commands, a zap-report/report.html should be generated in the pipeline artifacts, it contains the test results.

Getting hard disk info in mac using terminal

Kishan B — Tue, 16 Nov 2021 14:28:04 +0000

Its pretty easy. Just run the diskutil like the following

Command

diskutil info disk0

Output

   Device Identifier:         disk0
   Device Node:               /dev/disk0
   Whole:                     Yes
   Part of Whole:             disk0
   Device / Media Name:       APPLE SSD AP0256M

   Volume Name:               Not applicable (no file system)
   Mounted:                   Not applicable (no file system)
   File System:               None

   Content (IOContent):       GUID_partition_scheme
   OS Can Be Installed:       No
   Media Type:                Generic
   Protocol:                  PCI-Express
   SMART Status:              Verified

   Disk Size:                 251.0 GB (251000193024 Bytes) (exactly 490234752 512-Byte-Units)
   Device Block Size:         4096 Bytes

   Media OS Use Only:         No
   Media Read-Only:           No
   Volume Read-Only:          Not applicable (no file system)

   Device Location:           Internal
   Removable Media:           Fixed

   Solid State:               Yes
   Virtual:                   No

Configuring HP deskJet 2700 printer for printing over Wi-Fi

Kishan B — Mon, 27 Sep 2021 16:46:53 +0000

It was so frustrating for me to connect my printer(HP DeskJet Ink Advantage 2700 All-In-One printer) to print over wifi network and the most calming part in this frustrating experience is the slogan that is stamped on the printer! "Keep it simple" 🤣🤣

It is definitely not simple unless you know the trick 😊

Hence i am writing this blog for my future self and also people who face the same issue.

Instructions for a successful setup over Wi-Fi(using mobile app)

Ensure your mobile is connected to a Wi-Fi.
Install both HP smart and HP print service plugin mobile apps. If you are facing difficulties in finding these apps kindly visit https://123.hp.com
The most important secret is this, On the printer hold the Wireless button and cancel button for 3 seconds. The power button will blink for a few seconds, and the printer will be in setup mode(The wireless button will continue to blink until the printer is connected to the Wi-Fi network.)
Once this is done, add your printer as a new printer, in your mobile app and follow the steps in the apps. These steps apply even if you are changing from one wireless network to another. (Follow left to right in the image below)

Exposing custom prometheus metrics with dynamic labels in gin framework

Kishan B — Fri, 17 Sep 2021 10:55:26 +0000

We will be using golang, gin, prometheus go library to expose last request time metrics

Out of the box metrics

Spin up a gin go application which exposes the metrics which comes out of the box via the /metrics endpoint

package main

import (
    "github.com/gin-gonic/gin"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "log"
)

func main() {
    r := gin.Default()
    r.GET("/metrics", func(c *gin.Context) {
        handler := promhttp.Handler()
        handler.ServeHTTP(c.Writer, c.Request)
    })
    err := r.Run(":8080")
    if err != nil {
        log.Fatalln(err)
    }
}

Run the application using command go run main.go and
When you hit the /metrics you will see the following

$ curl http://localhost:8080/metrics

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 7
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1

(Truncated the above for brevity)

Custom last_request_received_time metric

Now let us expose our custom metric which is the last request time stamp

Going with gauge data type as for this use case we are "setting" the value as a timestamp

package main

import (
    "github.com/gin-gonic/gin"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "log"
)

func main() {
    r := gin.Default()

    // Gauge registration
    lastRequestReceivedTime := prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "last_request_received_time",
        Help: "Time when the last request was processed",
    })
    err := prometheus.Register(lastRequestReceivedTime)
    handleErr(err)

    // Middleware to set lastRequestReceivedTime for all requests
    r.Use(func(context *gin.Context) {
        lastRequestReceivedTime.SetToCurrentTime()
    })

    // Metrics handler
    r.GET("/metrics", func(c *gin.Context) {
        handler := promhttp.Handler()
        handler.ServeHTTP(c.Writer, c.Request)
    })
    err = r.Run(":8080")
    handleErr(err)
}

func handleErr(err error) {
    if err != nil {
        log.Fatalln(err)
    }
}

So now when you hit the metrics endpoint you will observe the newly created last_request_received_time metric.

$ curl http://localhost:8080/metrics

# HELP last_request_received_time Time when the last request was processed
# TYPE last_request_received_time gauge
last_request_received_time 1.63186694449664e+09

(removed the other metrics for brevity)

Custom last_request_received_time with dynamic labels

Now say we want to categorize this based on HTTP headers(Eg userId, Operation type). What this means is the label names are constant but the values are different.
For this case we have the GaugeVec type in the library

package main

import (
    "github.com/gin-gonic/gin"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "log"
    "net/http"
)

var (
    HeaderUserId        = "user_id"
    HeaderOperationType = "operation_type"
)

func main() {
    r := gin.Default()

    // Gauge registration
    lastRequestReceivedTime := prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "last_request_received_time",
        Help: "Time when the last request was processed",
    }, []string{HeaderUserId, HeaderOperationType})
    err := prometheus.Register(lastRequestReceivedTime)
    handleErr(err)

    // Metrics handler
    r.GET("/metrics", func(c *gin.Context) {
        handler := promhttp.Handler()
        handler.ServeHTTP(c.Writer, c.Request)
    })

    // Middleware to set lastRequestReceivedTime for all requests
    middleware := func(context *gin.Context) {
        lastRequestReceivedTime.With(prometheus.Labels{
            HeaderUserId:        context.GetHeader(HeaderUserId),
            HeaderOperationType: context.GetHeader(HeaderOperationType),
        }).SetToCurrentTime()
    }

    // Request handler
    r.GET("/data", middleware, func(c *gin.Context) {
        c.JSON(http.StatusOK, map[string]string{"status": "success"})
    })

    err = r.Run(":8080")
    handleErr(err)
}

func handleErr(err error) {
    if err != nil {
        log.Fatalln(err)
    }
}

$ curl -H "user_id: user1" -H "operation_type: fetch" http://localhost:8080/data
{"status":"success"}

$ curl -H "user_id: user1" -H "operation_type: view" http://localhost:8080/data
{"status":"success"}

$ curl -H "user_id: user1" -H "operation_type: upload" http://localhost:8080/data
{"status":"success"}

$ curl http://localhost:8080/metrics

# HELP last_request_received_time Time when the last request was processed
# TYPE last_request_received_time gauge
last_request_received_time{operation_type="fetch",user_id="user1"} 1.6318757060797539e+09
last_request_received_time{operation_type="upload",user_id="user1"} 1.631875691071805e+09
last_request_received_time{operation_type="view",user_id="user1"} 1.631875931726781e+09

As we see above for the same metric we have different valued labels

The code for this is available at
https://github.com/kishaningithub/lastrequesttimemetrics

Feedback is welcome :-)

References

Openssl by Example

Kishan B — Wed, 28 Jul 2021 06:11:17 +0000

Symmetric key
- Encryption
- Decryption
Asymmetric key
- Key Generation
  - Public key
  - Private key
- Encryption
- Decryption
- Sending signed messages
- Reading signed messages
- Encrypting private key
Others
- Find openssl version
- List ciphers

Symmetric key

Encryption

$ echo "top secret text" | openssl enc -aes-256-cbc -base64
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
<< encrypted text >>

Decryption

$ echo "<< encrypted text >>" | openssl enc -d -aes-256-cbc -base64
enter aes-256-cbc decryption password:
top secret text

Asymmetric key

Key Generation

Public key

openssl genrsa -out private.pem 2048

Private key

openssl rsa -in private.pem -pubout -out public.pem

Encryption

openssl rsautl -encrypt -in secret-transmission.txt -out secret-transmission.txt.enc -inkey public.pem -pubin

Decryption

openssl rsautl -decrypt -in secret-transmission.txt.enc -out secret-transmission.txt -inkey private.pem

Sending signed messages

openssl rsautl -sign -in secret-transmission.txt -out secret-transmission.txt.enc.signed -inkey private.pem

Reading signed messages

openssl rsautl -verify -in secret-transmission.txt.enc.signed -out secret-transmission.txt -inkey public.pem -pubin

Encrypting private key

Never store private key in clear text format!

openssl rsa -in private.pem -des3 -out private-enc.pem

Others

Find openssl version

$ openssl version
LibreSSL 2.6.4

List ciphers

openssl list-cipher-commands

DEV Community: Kishan B

Checking out code using github action in legacy runner

Problem

Solution

Note

How to use tfenv to install older versions of terraform in m1 mac?

How to get the DDL dump of a redshift database

Approach

Steps

Pros

Cons

Used TDD approach for a Hello World rust cli app

Problem

How did i do it?

Links

Use ZAP to Perform DAST (Dynamic Application Security Testing)

Introduction

What is Dynamic Application Security Testing (DAST)

What is ZAP

How to use ZAP

ZAP Scan for API

Local Run

Example - for API with Swagger

For SOAP/GraphQL

CircleCI Integrate

Example - for API with Swagger

For SOAP/GraphQL

ZAP Scan for Application (with UI)

Local Run for UI app

CircleCI Integration for UI app

Getting hard disk info in mac using terminal

Command

Output

Configuring HP deskJet 2700 printer for printing over Wi-Fi

Instructions for a successful setup over Wi-Fi(using mobile app)

Exposing custom prometheus metrics with dynamic labels in gin framework

Out of the box metrics

Custom last_request_received_time metric

Custom last_request_received_time with dynamic labels

References

Openssl by Example

Table of contents

Symmetric key

Encryption

Decryption

Asymmetric key

Key Generation

Public key

Private key

Encryption

Decryption

Sending signed messages

Reading signed messages

Encrypting private key

Others

Find openssl version

List ciphers