DEV Community: PGzlan

Identity and Access Management: Explained

PGzlan — Mon, 06 Nov 2023 05:34:19 +0000

The pillars of information security, namely Identification, Authentication, Authorization, and Accountability (IAAA), are crucial components in safeguarding sensitive information and resources by ensuring their confidentiality, integrity, and availability.

Identification refers to the process of verifying a user's claimed identity. Users provide a unique identifier, such as an email address, username, or ID number, to establish their identity within a particular system or environment. Many websites rely on email addresses as identifiers instead of requiring users to create unique usernames.
Authentication involves confirming that the user is indeed who they claim to be. It aims to validate the claimed identity. Passwords are a common authentication method, although other approaches like entering a code received via email are gaining popularity due to potential weaknesses in passwords.
Authorization determines the user's permitted access rights. It involves granting specific privileges and permissions based on the user's role, job function, or clearance level. By restricting access to only necessary resources, the risk of unauthorized access and data breaches is minimized.
Accountability entails tracking user activity to ensure they are held responsible for their actions. Once a user is granted access to a system, mechanisms are in place to record and store their activities in a centralized location. This logging of user activity serves as a means to identify the source of security incidents and take appropriate actions.

Identity Management (IdM)

Identity Management (IdM) is a vital aspect of cybersecurity that pertains to the management and regulation of digital identities. It encompasses the administration of user identities, including their authentication, authorization, and access control. The primary objective of IdM is to ensure that only authorized individuals can access specific resources and information. IdM systems are employed to oversee user identities across an organization's network.

IdM systems utilize a centralized database to store user identities and access privileges. These systems offer functionalities for managing and monitoring user access to resources. Common features found in IdM systems include user provisioning, authentication, and authorization. User provisioning involves the creation and administration of user accounts, while authentication and authorization involve verifying a user's identity and granting access to designated resources.

IdM systems play a critical role in organizations that utilize multiple systems and applications requiring access control. They streamline the management of user identities, mitigating the risk of unauthorized access to resources. Furthermore, IdM systems establish a central point of reference for user identity management, simplifying the process of managing user access rights for organizations.

Identity and Access Management (IAM)

IAM (Identity and Access Management) is a more inclusive concept compared to IdM (Identity Management). It encompasses a range of processes and technologies aimed at managing and safeguarding digital identities and access privileges. IAM systems incorporate various functions such as user provisioning, access control, identity governance, and compliance management. Their purpose is to ensure that only authorized individuals can access specific resources and data, while also monitoring and controlling their access.

IAM systems offer a comprehensive solution for managing and securing resource access within an organization. They integrate with multiple systems and applications, providing a centralized perspective on user identities and access rights. These systems utilize diverse technologies, including role-based access control, multi-factor authentication, and single sign-on, to manage access effectively.

IAM systems aid organizations in complying with regulatory requirements such as HIPAA, GDPR, and PCI DSS. They provide functionalities to handle the lifecycle of user identities, including onboarding, offboarding, and access revocation. Furthermore, IAM systems enable organizations to track and audit user activity, helping prevent security breaches and ensuring compliance with industry regulations.

Both IdM and IAM are integral components of cybersecurity as they ensure that only authorized individuals possess access to specific resources and information. While IdM systems focus on managing user identities, IAM systems encompass broader functions to manage and secure digital identities and access rights.

A system regulates access to different resources using a selected model. Several prevalent access control models include:

Discretionary Access Control (DAC)
Role-Based Access Control (RBAC)
Mandatory Access Control (MAC)
Attribute-Based Access Control (ABAC)

In these models, access to resources is managed and granted based on different principles and mechanisms.

Discretionary Access Control

Many people have utilized Discretionary Access Control (DAC) to share files or folders with friends and colleagues. In DAC, the owner of the resource explicitly adds users and assigns them appropriate permissions.

For example, let's say you have your photos stored on an online platform. To share the images related to your graduation with your family, you individually add their accounts and grant them access to the specific album. As a result, the album permissions will reflect a few accounts with viewing privileges.

This process is straightforward and under the complete control of the data owner. It works well for sharing with family members or a select few users within a company. However, difficulties can arise when attempting to scale sharing to a larger number of users, particularly as users' roles evolve over time. This situation leads us to consider sharing based on user roles.

Role-Based Access Control

Role-Based Access Control (RBAC) employs a highly intuitive approach. Each user is assigned one or more roles or functional positions, which determine their authorized access to different resources.

For example, an accountant may require access to the company's accounting books but does not need access to research and development labs or documents. Consequently, users are categorized into various groups based on their roles. Authorization and access are then granted based on the group to which a user belongs.

The classification of users according to their roles offers numerous advantages. For instance, when a user assumes a new role, it is simply a matter of adding them to the corresponding group. Similarly, if a user relinquishes a particular role, they can be removed from the relevant group. This approach greatly simplifies and streamlines maintenance tasks, making them more efficient.

Mandatory Access Control

An operating system that implements Mandatory Access Control (MAC) prioritizes security and imposes significant restrictions on user capabilities. Such systems are typically used for specific purposes or to handle highly classified data. Consequently, users are limited to performing only essential tasks, and they are unable to carry out actions like installing new software or modifying file permissions.

AppArmor is a tool that enables the implementation of MAC on Linux distributions. It is already included in various Linux distributions such as Debian and Ubuntu.

The SELinux project offers a flexible MAC solution for Linux systems. It is a standard feature in several Linux distributions like Red Hat and Fedora.

Attribute-Based Access Control

Attribute-Based Access Control (ABAC) is a sophisticated access control model that grants permissions based on attributes associated with users, resources, and environmental conditions. Unlike traditional access control models that rely on roles or groups, ABAC considers a wide range of attributes in its decision-making process. These attributes can include user characteristics, such as job title or department, resource properties, such as sensitivity or location, and contextual factors, such as time of day or network location.

By evaluating these attributes, ABAC policies can dynamically determine whether to grant or deny access to specific resources. This dynamic decision-making capability allows organizations to implement more fine-grained access control policies that align with their unique security requirements. For example, an organization may define a policy stating that only employees with a job title of "Manager" can access confidential financial data during business hours, while restricting access for other employees.

ABAC provides a flexible and granular approach to access control. Organizations can define complex policies that take into account multiple attributes and their combinations to make access control decisions. This flexibility allows for the implementation of more nuanced and context-aware access control policies. Additionally, ABAC supports the concept of dynamic and adaptive authorization, enabling real-time adjustments to access permissions based on changing attributes or environmental conditions.

Implementing ABAC typically involves the use of an attribute-based policy language and an attribute-based policy enforcement point. The policy language allows organizations to define and express their access control policies using attributes and logical expressions. The policy enforcement point evaluates these policies at runtime and enforces access decisions based on the attributes associated with the requesting user, requested resource, and the environment.

Python's IceCream Package: Effortless Debugging with Minimal Code

PGzlan — Sun, 29 Oct 2023 22:51:33 +0000

Introduction

Often times when programming in Python, things don't work the way we expect them to work. We implement a function, and the function does not deliver the results that we were expect from it. Well, at least not all the time, and we have no clear reason to identify this behavior. Imagine some function taking some parameters param1...paramN, then these parameters are used in a loop, which contains some conditional statements. So we have ourselves a complicated function, with loops, and conditions and things can become rather unpredictable.

Of course the proper way to debug such a function would be by leveraging a debugger in your IDE (Integrated Development Environment) or your code editor by setting up break points to check what modifications occur to the variables inside. This can be rather tedious and a lot of time you find yourself avoiding the debugger and debugging mode. We just want to execute the code as it is right now, with a little bit of additional information so what ends up happening is just using good ol' print statements.

Debugging is an essential part of software development, and print statements have long been used to debug code. However, traditional print statements can be tedious to write and can clutter code. That's where the icecream (ic) package comes in. IceCream is a Python library that makes debugging effortless and readable with minimal code [1]. In this blog post, we will explore the features of the IceCream package and provide code examples to demonstrate its usage.

Usage

Let's say as a simple example, we have some function multiply that takes a and b as parameters and returns the simple value of a * b



from icecream import ic

def multiply(a, b):
    return a * b

To test this function, we can write it in the following manner



print(multiply(2, 3))
print(multiply(3, 4))
print(multiply(4, 5))
print(multiply(5, 6))

The output here would be:

This seems relatively simple to trace. That would not hold in cases we have a longer list of values to test. If this is a more complex function, we cannot predict immediately by looking at the function call what the result is going to. It would be nicer to have the function call to what is actually being called and what is the result of this function call and this is exactly what icecream ic does [2].



ic(multiply(2, 3))
ic(multiply(3, 4))
ic(multiply(4, 5))
ic(multiply(5, 6))

The output will look like this:



ic| multiply(2, 3): 6
ic| multiply(3, 4): 12
ic| multiply(4, 5): 20
ic| multiply(5, 6): 30

One of the neat things about ic is, unlike print (which have a None return value), using ic will not only provide a logging message, it will also get the returned value stored into the variable you assign it to:



res = ic(multiply(5, 6))
print(res)

In this case, the output would be:



30
ic| multiply(5, 6): 30

Multiple Values

You can also use ic() to print multiple values at once. Here's an example:



from icecream import ic

def multiply_and_divide(a, b):
    result = a * b
    ic(result, a/b)
    return result, a/b

multiply_and_divide(2, 3)

In this example, we're using ic() to print the result of the multiplication operation and the result of the division operation. The output will look like this:



ic| result: 6, a/b: 0.6666666666666666

Note that ic() automatically separates the values with commas in the output [2].

Conditional Statements

You can use ic() in conditional statements to print values only when certain conditions are met. Here's an example:



from icecream import ic

def is_even(num):
    result = num % 2 == 0
    ic(result, num)
    return result

is_even(4)
is_even(5)

In this example, we're using ic() to print the result of the conditional statement and the input value only when the input value is even. The output will look like this:



ic| result: True, num: 4
ic| result: False

Note that ic() will only print the second argument (num) when the first argument (result) is True [2].

Iterables

You can use ic() to print the contents of iterables such as lists, tuples, and dictionaries. Here's an example:



from icecream import ic

def sum_list(numbers):
    result = sum(numbers)
    ic(numbers, result)
    return result

sum_list([1, 2, 3, 4, 5])

In this example, we're using ic() to print the input list and the sum of the list. The output will look like this:



ic| numbers: [1, 2, 3, 4, 5], result: 15

Note that ic() automatically prints the contents of the list [2].

Functions

You can use ic() to print the result of a function call. Here's an example:



from icecream import ic

def multiply(a, b):
    result = a * b
    return result

def square_and_multiply(a, b):
    result = multiply(a**2, b)
    ic(result)
    return result

square_and_multiply(2, 3)

In this example, we're using ic() to print the result of the multiply function call within the square_and_multiply function. The output will look like this:



ic| result: 12

Note that ic() automatically prints the return value of the function [2].

Nifty things about `ic`

Using ic provides some nice utility in terms of readability, especially for large dictionaries. Let's say we have the following dict:



data = {
  "contacts": [
    {
      "name": "John Doe",
      "contactMethods": {
        "phone": "123-456-7890",
        "email": "john.doe@example.com"
      }
    },
    {
      "name": "Jane Smith",
      "contactMethods": {
        "phone": "987-654-3210",
        "email": "jane.smith@example.com"
      }
    },
    {
      "name": "Robert Johnson",
      "contactMethods": {
        "phone": "555-123-4567",
        "email": "robert.johnson@example.com"
      }
    },
    {
      "name": "Emily Davis",
      "contactMethods": {
        "phone": "111-222-3333",
        "email": "emily.davis@example.com"
      }
    },
    {
      "name": "Michael Brown",
      "contactMethods": {
        "phone": "444-555-6666",
        "email": "michael.brown@example.com"
      }
    }
  ]
}

We can print it using ic to get some nice indentation and color

If we call ic() at a certain part of the function like this:



from icecream import ic

def multiply_and_divide(a, b):
    result = a * b
    ic()
    return result, a/b

multiply_and_divide(2, 3)

The output will look like this:



ic| <ipython-input-13-5d4984d3accb>:5 in multiply_and_divide() at 21:50:10.905
Out[13]: (6, 0.6666666666666666)

This can be useful to determine where far we have reached in a certain function. You can also enable or disable it selectively



multiply_and_divide(2, 3)

multiply_and_divide(4, 5)

ic.disable()

multiply_and_divide(5, 6)

ic.enable()

multiply_and_divide(6, 7)



ic| <ipython-input-13-5d4984d3accb>:5 in multiply_and_divide() at 21:59:35.824

ic| <ipython-input-13-5d4984d3accb>:5 in multiply_and_divide() at 21:59:35.826

ic| <ipython-input-13-5d4984d3accb>:5 in multiply_and_divide() at 21:59:35.828

Out[14]: (42, 0.8571428571428571)

Conclusion

IceCream is a powerful tool for debugging Python code that replaces the traditional print statement. It provides detailed information about variables, expressions, and function calls that make it easy to identify and fix errors in your code. With its customizable output, logging capabilities, and ability to access external modules, IceCream is a versatile debugging tool that can save you time and frustration. By using the examples provided in this blog post, you can start using IceCream in your own Python projects and improve your debugging workflow.

References

Python Walrus Operator: An Introduction

PGzlan — Sat, 28 Oct 2023 19:14:20 +0000

Python 3.8 introduced a new operator called the walrus operator, denoted by :=. It's called the Walrus operator because it looks like the eyes and the tusks of a walrus on its side. This operator allows you to evaluate an expression and assign its value to a variable in a single statement.

What is the Walrus Operator?

The walrus operator, also known as the assignment expression operator, is a syntax enhancement that has been introduced in Python 3.8. It allows you to assign a value to a variable within an expression. The syntax for the walrus operator is (variable := expression) [1].

Basic Usage

Let's start with a basic example to understand how the walrus operator works. Consider the following code snippet:

walrus = (value := 3357)
print(f"walrus: {walrus}")
print(f"value: {value}")

Output:

walrus: 3357
value: 3357

In this example, we use the walrus operator to define the variable value and assign it the value 3357. The expression (value := 3357) returns the assigned value, which we then use to initialize the variable walrus. As a result, both variables have the same value [1].

Use Cases

1. If Statements

The walrus operator can be useful in if statements when you want to perform an operation on an unknown variable. For example, let's say you want check if a specific environment variable is set and based on that, you want to execute a set of actions. This is one way it could be done conventionally:

import os
import shutil

def delete_folder_contents():
    # Get the value of the "PGDATA" environment variable
    pgdata_path = os.environ.get("PGDATA", None)

    if pgdata_path:
        # Verify if the path exists
        if os.path.exists(pgdata_path):
            # Iterate over the files and folders within the directory
            for filename in os.listdir(pgdata_path):
                file_path = os.path.join(pgdata_path, filename)

                # Check if it is a file and delete it
                if os.path.isfile(file_path):
                    os.remove(file_path)
                # Check if it is a directory and delete it recursively
                elif os.path.isdir(file_path):
                    shutil.rmtree(file_path)

            print("Folder contents deleted successfully.")
        else:
            print("The specified path does not exist.")
    else:
        print("The 'PGDATA' environment variable is not set.")

delete_folder_contents()

Here's how you can achieve this using the walrus operator:

import os
import shutil

def delete_folder_contents():
    # Verify if the path exists
    if (pgdata_path := os.environ.get("PGDATA", None)) is not None:
        # Check if the path exists
        if os.path.exists(pgdata_path):
            # Iterate over the files and folders within the directory
            for filename in os.listdir(pgdata_path):
                file_path = os.path.join(pgdata_path, filename)

                # Check if it is a file and delete it
                if os.path.isfile(file_path):
                    os.remove(file_path)
                # Check if it is a directory and delete it recursively
                elif os.path.isdir(file_path):
                    shutil.rmtree(file_path)

            print("Folder contents deleted successfully.")
        else:
            print("The specified path does not exist.")
    else:
        print("The 'PGDATA' environment variable is not set.")

delete_folder_contents()

The assignment expression (pgdata_path := os.environ.get("PGDATA", None)) assigns the value of os.environ.get("PGDATA", None) to pgdata_path and simultaneously checks if it is not None. This allows you to combine the assignment and the if statement in a single line, removing one level of indentation.

2. For Loops

The walrus operator can also be used in for loops. Consider the following example where we want to create a list of squares of numbers in the input list if the squares are greater than 100:

def fibonacci(n):
    if 0 <= n < 2:
        return n
    elif n < 0:
        raise ValueError("Negative Fibonnaci not supported")
    else:
        fib_seq = (0, 1)    
        for i in range(n - 1):
            (fib_seq := (fib_seq[1], sum(fib_seq)))    
    return fib_seq[1]

n = 10
fib_sequence = fibonacci(n)
print(f"The Fibonacci sequence in the position {n} is: {fib_sequence}")

Output:

The Fibonacci sequence in the position 10 is: 55

In this example, the walrus operator (fib_seq := (fib_seq[1], sum(fib_seq))) sets the values of the tuple (moves the second item in first position and places the sum of the existing values in the second position) and assigns that tuple to the variable fib_seq

3. List Comprehension

The walrus operator can also be used in list comprehensions to simplify the code. Consider the following example where we want to create a new list of squares of numbers from the input list of prime numbers if the squares are greater than 100:

prime_list = [2, 3, 5, 7, 11]
applicable_list = [psqrd for prime in prime_list if (psqrd := prime ** 2) >= 100]
print("The original list is:", prime_list)
print("The output list is:", applicable_list)

Output:

The original list is: [2, 3, 5, 7, 11]
The output list is: [121]

In this example, the walrus operator (psqrd := prime ** 2) calculates the square of each number in the input list and assigns it to the variable psqrd. If the square is greater than or equal to 100, it is added to the applicable_list [1].

4. While Loops

The walrus operator can also be used in while loops. Consider the following example where we want to read input from a user until they enter the flag \q'. One way of doing it would be:

data = []
while True:
    d = input('Please provide the required data: ')
    if d == '\q':
        break
    data.append(d)
print(f'Data passed is: {data}')

Output:

Please provide the required data: 1
Please provide the required data: 2
Please provide the required data: 3
Please provide the required data: \q
Data passed is: ['1', '2', '3']

We can modify this example to use the walrus operator to simplify the loop by creating the variable passed to input as well as check its value:

data = []
while (d := input('Please provide the required data: ')) != '\q':
    data.append(d)
print(f'Data passed is: {data}')

Output:

Please provide the required data: 1
Please provide the required data: 2
Please provide the required data: 3
Please provide the required data: \q
Data passed is: ['1', '2', '3']

Controversy Surrounding the Walrus Operator

Like any new feature, the walrus operator has sparked some controversy within the Python community. Some developers argue that it violates certain heuristics and can lead to less readable code [2]. However, others find it to be a powerful addition that simplifies certain programming tasks.

Conclusion

The walrus operator might just be a valuable addition to Python's syntax, providing a concise and efficient way to assign and evaluate expressions. While it may be a subject of controversy, it offers developers new possibilities for writing clean and efficient code.

Resources

Exploring Different Types of Joins in Pandas

PGzlan — Mon, 23 Oct 2023 08:34:48 +0000

When working with data in Python, the Pandas library provides powerful tools for data manipulation and analysis. One essential operation when working with multiple datasets is joining them based on common columns. Pandas offers several types of joins to combine data efficiently. In this blog post, we will explore the different types of joins in Pandas and provide code examples using some dummy data.

Understanding Joins

A join operation combines two or more datasets based on a common column or index. The resulting dataset includes rows that have matching values in the specified columns or indices. Pandas provides several join methods, each with its own behavior and use cases. Let's take a closer look at each one.

Inner Join

An inner join returns only the rows that have matching values in both datasets. In other words, it keeps only the intersection of the two datasets. To demonstrate this, let's consider two dummy datasets:

import pandas as pd

# Create dummy datasets
df1 = pd.DataFrame({'ID': [1, 2, 3], 'Value': ['A', 'B', 'C']})
df2 = pd.DataFrame({'ID': [2, 3, 4], 'Data': [10, 20, 30]})

# Perform an inner join
inner_join = pd.merge(df1, df2, on='ID', how='inner')
print(inner_join)

The pd.merge() function is used to perform the join. By specifying how='inner', we perform an inner join based on the 'ID' column. The result will contain only the rows with matching IDs in both datasets. In this case, the output will be:

   ID Value  Data
0   2     B    10
1   3     C    20

Left Join

A left join returns all the rows from the left dataset and the matched rows from the right dataset. If no match is found in the right dataset, the corresponding values will be filled with NaN (missing values). Let's illustrate this with an example:

# Perform a left join
left_join = pd.merge(df1, df2, on='ID', how='left')
print(left_join)

The left join keeps all the rows from the left dataset (df1) and adds the matching rows from the right dataset (df2). If there is no match, the 'Data' column will have NaN values. The output will be:

   ID Value  Data
0   1     A   NaN
1   2     B  10.0
2   3     C  20.0

Right Join

A right join is similar to a left join but keeps all the rows from the right dataset and the matched rows from the left dataset. If no match is found in the left dataset, the corresponding values will be NaN. Let's see it in action:

# Perform a right join
right_join = pd.merge(df1, df2, on='ID', how='right')
print(right_join)

The right join includes all the rows from the right dataset (df2) and adds the matching rows from the left dataset (df1). If there is no match, the 'Value' column will contain NaN values. The output will be:

   ID Value  Data
0   2     B    10
1   3     C    20
2   4   NaN    30

Outer Join

An outer join combines all the rows from both datasets. If a row has no match in the other dataset, the corresponding values will be NaN. This join type is useful when you want to retain all the information from both datasets. Let's try an outer join:

# Perform an outer join
outer_join = pd.merge(df1, df2, on='ID', how='outer')
print(outer_join)

The outer join returns all rows from both datasets, filling missing values with NaN. The output will be:

   ID Value  Data
0   1     A   NaN
1   2     B  10.0
2   3     C  20.0
3   4   NaN  30.0

Conclusion

Joining datasets is a fundamental operation when working with multiple sources of data. In this blog post, we explored the different types of joins available in the Pandas library. We covered the inner join, left join, right join, and outer join, explaining their behaviors and providing code examples with dummy data.

By understanding and using these join methods effectively, you can combine data from different sources to gain valuable insights and perform comprehensive data analysis. The ability to merge datasets based on common columns or indices is a powerful tool in your data manipulation toolkit.

Remember to consider your specific use case and the relationships between your datasets when choosing the appropriate join method. Each join type has its advantages and is suitable for different scenarios. Experimentation and practice will help you become more proficient in using joins effectively.

Pandas provides a wide range of functionalities for data manipulation, and joins are just one aspect of its capabilities. As you delve deeper into data analysis and manipulation, you'll discover even more powerful features and techniques offered by the library.

References

MITRE ATT&CK: Your map to adversary movement

PGzlan — Sun, 15 Oct 2023 23:40:48 +0000

Introduction to MITRE ATT&CK
The Framework
1. Tactics
2. Techniques
3. Sub-techniques
4. Procedures
How MITRE ATT&CK Works
1. Adversary Emulation
2. Threat Intelligence
The Benefits of MITRE ATT&CK
1. Improved Threat Detection
2. Enhanced Incident Response
3. Better Security Assessments
Implementing MITRE ATT&CK
1. Mapping Techniques and Controls
2. Leveraging MITRE ATT&CK for Red Teaming
Limitations and Challenges
Conclusion

1. Introduction to MITRE ATT&CK

In today's rapidly evolving threat landscape, organizations face constant challenges in defending against sophisticated cyber attacks. To address this, MITRE Corporation developed the MITRE ATT&CK framework. ATT&CK, which stands for Adversarial Tactics, Techniques, and Common Knowledge, is a globally accessible knowledge base of adversary tactics and techniques. It provides a structured and comprehensive approach to understanding and categorizing cyber threats.

2. The Framework

The MITRE ATT&CK framework is organized hierarchically, consisting of tactics, techniques, and sub-techniques. This structure allows security professionals to gain a deep understanding of how adversaries operate and the tactics they employ.

2.1. Tactics

The tactics within the ATT&CK framework represent the underlying purpose or objective (the why) behind each technique or sub-technique. They define the technical goals that adversaries aim to accomplish and the rationale behind their actions. For instance, an adversary might target credential access as a means to gain entry into a targeted network.

Each tactic encompasses a set of techniques that have been observed by network defenders in real-world scenarios involving threat actors. It's important to note that the ATT&CK framework is not meant to be interpreted as a linear progression, where adversaries move through tactics in a straightforward manner from left to right to achieve their objectives. Furthermore, adversaries are not obligated to utilize all ATT&CK tactics in order to accomplish their operational goals.

The MITRE ATT&CK framework defines several tactics, including the following:

Initial Access
Execution
Persistence
Privilege Escalation
Defense Evasion
Credential Access
Lateral Movement
Exfiltration
Impact

2.2. Techniques

Techniques within the context of the ATT&CK framework describe the (how) methods employed by adversaries to accomplish a specific tactical objective through their actions. For instance, an adversary might employ the technique of dumping credentials to gain access to privileged accounts.

Techniques can also indicate what an adversary acquires as a result of their actions. A technique represents a distinct behavior aimed at achieving a particular goal and is often just one step in a series of activities carried out by the adversary to fulfill their overall mission.

It should be noted that certain techniques within ATT&CK can enable an adversary to achieve multiple tactical objectives, thus applying to multiple tactics within the framework.

Additionally, many techniques involve the utilization of legitimate system functions that can be exploited for malicious purposes, commonly known as "living off the land." (at.exe, schdtsk, etc)

2.3. Sub-techniques

Sub-techniques provide further granularity within each technique, allowing for a more detailed understanding of adversary behaviors. They describe specific variations or methods used by adversaries to accomplish a technique within a tactic. Sub-techniques help in capturing the nuances and evolving tactics employed by adversaries.

For instance, within the brute force T1110 technique, there are specific behaviors that outline distinct methods for executing the technique. These behaviors include password guessing T1110.001, password cracking T1110.002, password spraying T1110.003, and credential stuffing T1110.004

Sub-techniques are often associated with specific operating systems or platforms, although this is not always the case. It's important to note that not all techniques have corresponding sub-techniques.

2.4. Procedures

Procedures are indicators that represent the specific actions (the what) performed by an adversary and serve as examples of how they have utilized a technique or sub-technique. For instance, when it comes to OS credential dumping, there exist various procedures like the extraction of LSASS memory T1003.001 using different tools, utilities, and commands.

Understanding these procedures can be valuable for replicating an incident through adversary emulation and for detecting malicious activity.

3. How MITRE ATT&CK Works

MITRE ATT&CK works by providing organizations with a knowledge base that can be used to improve their cybersecurity defenses. It offers two primary components:

3.1. Adversary Emulation

Adversary emulation involves simulating real-world attack scenarios in a controlled environment to evaluate an organization's security controls and incident response capabilities. By replicating adversary behaviors and techniques, organizations can identify vulnerabilities and improve their defenses.

3.2. Threat Intelligence

MITRE ATT&CK provides valuable threat intelligence by documenting the tactics, techniques, and procedures (TTPs) used by real-world adversaries. This information helps security teams understand the latest trends in cyber attacks, identify potential indicators of compromise (IOCs), and develop effective countermeasures.

4. The Benefits of MITRE ATT&CK

4.1. Improved Threat Detection

By aligning security controls and detection mechanisms with the MITRE ATT&CK framework, organizations can enhance their ability to detect and respond to advanced threats. The framework provides a standardized language and taxonomy for understanding adversary behaviors, enabling more effective threat hunting and incident response.

4.2. Enhanced Incident Response

MITRE ATT&CK helps organizations develop robust incident response plans by providing insights into the tactics and techniques employed by adversaries. This knowledge allows for more efficient and targeted incident response efforts, minimizing the impact of cyber attacks and reducing recovery time.

4.3. Better Security Assessments

By leveraging MITRE ATT&CK, organizations can perform comprehensive security assessments to identify vulnerabilities and gaps in their defenses. The framework offers a standardized methodology for evaluating security controls, enabling organizations to prioritize remediation efforts and allocate resources effectively.

5. Implementing MITRE ATT&CK

5.1. Mapping Techniques and Controls

Organizations can map their existing security controls to the techniques and tactics defined in the MITRE ATT&CK framework. This mapping helps identify coverage gaps and areas where additional controls are needed to mitigate specific threats. It provides a proactive approach to improve security postures.

5.2. Leveraging MITRE ATT&CK for Red Teaming

Red teaming exercises involve emulating real-world attacks to test an organization's defenses. By using the MITRE ATT&CK framework, red teams can align their tactics and techniques with known adversary behaviors, providing a more realistic assessment of an organization's security posture.

6. Limitations and Challenges

While MITRE ATT&CK is a valuable resource, it is not without limitations. The framework requires continuous updates to keep pace with evolving adversary tactics. It also requires organizations to invest time and resources in understanding and implementing the framework effectively.

7. Conclusion

MITRE ATT&CK offers a comprehensive and structured approach to understanding and categorizing adversary behaviors. By leveraging the framework, organizations can enhance their threat detection, incident response, and overall security posture. However, it is important to recognize the limitations and challenges associated with implementing and maintaining MITRE ATT&CK effectively.

Note: This document is a summary based on current understanding and does not represent an actual MITRE ATT&CK guide.

Using Shared Memory in Linux

PGzlan — Sat, 30 Sep 2023 19:50:47 +0000

Introduction:

Shared memory is a powerful feature in Linux that allows processes to communicate and share data efficiently. In this blog post, we will delve into the world of shared memory, focusing on important functions like shm_open. We will explore how shared memory works, discuss relevant functions, provide a code example, and highlight the benefits of using shared memory for inter-process communication. Additionally, we will address potential security risks associated with shared memory, particularly the concern of data being read off disk. By the end of this post, you will have a comprehensive understanding of shared memory in Linux, enabling you to leverage its advantages while ensuring data security.

Introduction to Shared Memory
Shared Memory Functions
- shm_open
- shm_unlink
- shmget and shmat
- shmdt
Benefits of Using Shared Memory
Security Risks: Data Read Off Disk
- Understanding the Vulnerability
- Mitigating Risks
Code Example
Conclusion
References

1. Introduction to Shared Memory

Shared memory allows multiple processes to access the same region of memory, facilitating fast and efficient communication between them. Instead of using slower inter-process communication methods like pipes or sockets, shared memory enables direct access to shared data, resulting in improved performance.

2. Shared Memory Functions

Several key functions are used to work with shared memory in Linux. Let's explore them in detail.

`shm_open`

The shm_open function creates or opens a shared memory object. It takes a name and flags as arguments and returns a file descriptor that can be used to access the shared memory.

`shm_unlink`

The shm_unlink function removes a shared memory object by name. It ensures that the shared memory is properly cleaned up after use.

int shm_unlink(const char *name);

`shmget` and `shmat`

The shmget function allocates a new shared memory segment or retrieves an existing one based on a key. On the other hand, shmat attaches the shared memory segment to the address space of the calling process.

int shmget(key_t key, size_t size, int shmflg);
void *shmat(int shmid, const void *shmaddr, int shmflg);

`shmdt`

The shmdt function detaches a shared memory segment from the address space of the calling process.

int shmdt(const void *shmaddr);

3. Benefits of Using Shared Memory

Using shared memory offers several advantages:

Speed: Shared memory allows for faster data exchange between processes since it avoids the overhead of copying data.
Efficiency: Processes can directly access shared data, eliminating the need for serialization and deserialization.
Synchronization: Shared memory can be combined with synchronization mechanisms like semaphores ormutexes to ensure proper access control.
Resource Sharing: Shared memory allows processes to share large amounts of data without duplicating it in memory.

4. Security Risks: Data Read Off Disk

One potential security concern with shared memory is the possibility of sensitive data being read off disk. Shared memory objects are backed by files in the file system, and if the system crashes, the data may be temporarily stored on disk.

Understanding the Vulnerability

When data resides on disk, it becomes vulnerable to unauthorized access. Malicious users or processes with sufficient privileges may attempt to read the shared memory files directly, potentially compromising sensitive information.

Mitigating Risks

To mitigate the security risks associated with shared memory, consider the following best practices:

Use Appropriate Permissions: Set appropriate file permissions for the shared memory files to restrict access to authorized users and processes.
Encrypt Sensitive Data: If the shared memory contains sensitive information, consider encrypting the data before storing it in the shared memory segment.
Secure the System: Implement strong security measures on the system, such as access controls, to prevent unauthorized access to shared memory files.
Handle Segmentation Faults: Properly handle segmentation faults to prevent sensitive data from being inadvertently written to disk.

By following these best practices, you can minimize the potential security risks associated with shared memory usage.

5. Code Example

Here's a code example that demonstrates the usage of shared memory in Linux:

// Include the necessary headers

int main() {
    const char *name = "/my_shared_memory";
    int shm_fd = shm_open(name, O_CREAT | O_RDWR, 0666);
    if (shm_fd == -1) {
        perror("shm_open");
        return 1;
    }

    // Set the size of the shared memory region
    size_t size = 1024;

    // Resize the shared memory object to the desired size
    if (ftruncate(shm_fd, size) == -1) {
        perror("ftruncate");
        return 1;
    }

    // Map the shared memory object into the process address space
    void *ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, shm_fd, 0);
    if (ptr == MAP_FAILED) {
        perror("mmap");
        return 1;
    }

    // Write data to the shared memory
    sprintf(ptr, "Hello, shared memory!");

    // Print the contents of the shared memory
    printf("Shared memory contents: %s\n", (char *)ptr);

    // Unmap the shared memory object
    if (munmap(ptr, size) == -1) {
        perror("munmap");
        return 1;
    }

    // Close the shared memory file descriptor
    if (close(shm_fd) == -1) {
        perror("close");
        return 1;
    }

    // Unlink the shared memory object
    if (shm_unlink(name) == -1) {
        perror("shm_unlink");
        return 1;
    }

    return 0;
}

This code example demonstrates creating a shared memory object using shm_open. We allocate a shared memory segment of size 1024 bytes using ftruncate. Then, we map the shared memory into the process address space using mmap and write a string to it. Finally, we unmap the shared memory, close the file descriptor, and unlink the shared memory object.

6. Conclusion

Shared memory provides a powerful mechanism for efficient inter-process communication in Linux. Through functions like shm_open, shm_unlink, shmget, and shmat, developers can leverage shared memory to enhance performance and resource sharing among processes. However, it is crucial to be aware of the security risks associated with shared memory, particularly the possibility of data being read off disk. By implementing best practices and taking appropriate security measures, developers can ensure the integrity and confidentiality of shared data.

In this blog post, we explored the concept of shared memory in Linux, understanding its benefits and the relevant functions such as shm_open, shm_unlink, shmget, and shmat. We also discussed the potential security risks associated with shared memory, emphasizing the importance of securing shared memory files and implementing strong access controls.

Shared memory remains a valuable tool for efficient and high-performance inter-process communication. By understanding its functionality and taking necessary precautions, developers can harness the power of shared memory while safeguarding sensitive data.

Remember to always prioritize security when working with shared memory and stay vigilant in protecting the confidentiality and integrity of shared data. With the right knowledge and practices, shared memory can be a reliable and effective solution for seamless communication and collaboration among processes in Linux.

References

Efficiency and Speed with GNU Parallel

PGzlan — Sat, 30 Sep 2023 19:50:20 +0000

Introduction
Understanding GNU Parallel
Installation and Basic Usage
Parallelization Modes and Job Control
Advanced Features and Examples
Advantages of Using GNU Parallel
Conclusion

Introduction

In the world of command-line utilities, GNU Parallel stands out as a powerful tool for achieving parallelism and optimizing computational tasks. This article explores the fundamental concepts of GNU Parallel, demonstrates its usage through practical examples, and highlights the advantages it offers in terms of efficiency, scalability, and flexibility.

1. Understanding GNU Parallel

GNU Parallel is a command-line tool designed to execute tasks in parallel, unlocking the potential of modern multi-core processors and distributed computing environments. It allows users to divide a workload into smaller units (jobs) and run them simultaneously, thereby significantly reducing the overall execution time. By default, GNU Parallel distributes jobs across available CPU cores, but it can also leverage remote servers to achieve distributed computing.

2. Installation and Basic Usage

To get started with GNU Parallel, it must first be installed on your system. On Linux, you can typically install it using the package manager of your distribution. Once installed, GNU Parallel can be used from the command line.
The basic syntax for running commands in parallel with GNU Parallel is:

parallel [options] command ::: arguments

Here, command represents the task or command to be executed, and arguments are the inputs passed to the command. Multiple arguments can be specified, separated by spaces or provided through a file.
For example, to display a list of files in parallel using the cat command , the following command can be used:

parallel cat::: file1.txt file2.txt file3.txt

GNU Parallel automatically distributes the files across available CPU cores, executing the echo command concurrently.

3. Parallelization Modes and Job Control

GNU Parallel offers different modes of parallelization to suit various scenarios. It supports both local parallelization and remote execution across multiple machines.
Additionally, GNU Parallel provides fine-grained control over job execution. For instance, the --dry-run option allows users to preview the commands without actually executing them. Furthermore, the --halt option enables the termination of parallel execution if any job fails, ensuring robustness in complex workflows.

4. Advanced Features and Examples

GNU Parallel offers a plethora of advanced features that make it a versatile tool for data processing and beyond.

a. Input Sources:

GNU Parallel can read input data from various sources, including command substitution, standard input (stdin), files, or even environment variables. This flexibility allows seamless integration with other command-line tools and pipelines. For example:

cat files.txt | parallel process_file

b. Command Substitution:

Commands can be generated dynamically using command substitution within GNU Parallel. This enables the execution of complex workflows and the passing of dynamically generated arguments to commands. For example:

parallel "process_{}" ::: $(ls *.txt)

c. Progress Monitoring:

Using the --progress option, GNU Parallel provides real-time progress monitoring, displaying job execution status, completion percentage, and ETA (Estimated Time of Arrival). This feature is particularly useful when dealing with time-consuming tasks. For example:

parallel --progress process_file ::: *.txt

d. Load Balancing:

GNU Parallel supports load balancing mechanisms, ensuring equal distribution of jobs across processors or machines. It can automatically adjust the workload distribution based on factors such as job completion time or system load, optimizing resource utilization.

5. Advantages of Using GNU Parallel

The utilization of GNU Parallel brings several advantages to data processing and computational tasks:

a. Improved Efficiency:

By harnessing the power of parallel computing, GNU Parallel significantly reduces the execution time of time-consuming tasks, leading to enhanced efficiency and productivity.

b. Scalability and Flexibility:

GNU Parallel effortlessly scales from single-core systems to multi-core machines or even distributed computing environments, enabling seamless parallelization across different setups.

c. Simplified Parallelization:

GNU Parallel abstracts the complexities of parallel computing, allowing users to focus on task definition rather than intricate parallelization mechanisms. This simplification accelerates development and facilitates code maintenance.

d. Resource Optimization:

With load balancing and job control features, GNU Parallel optimizes resource utilization by distributing jobs evenly and monitoring their execution. This ensures efficient utilization of CPU cores and minimizes idle times.

e. Seamless Integration:

GNU Parallel seamlessly integrates with existing workflows, as it can be combined with other command-line tools and utilities. This interoperability enables users to leverage the full potential of their existing tools while achieving parallel execution.

6. Conclusion

GNU Parallel empowers users to unlock the potential of parallel computing, enabling faster and more efficient execution of computational tasks. By distributing jobs across available CPU cores or remote servers, GNU Parallel revolutionizes data processing and computational workflows, offering scalability, simplicity, and resource optimization. With its easy-to-use syntax and advanced features, GNU Parallel is a valuable addition to any command-line toolkit, providing substantial speed improvements and unlocking the full potential of modern computing systems.

Git Patches: Your friend to less Copy-Paste

PGzlan — Sun, 24 Sep 2023 21:40:56 +0000

Git, the most widely used version control system, offers a powerful feature called patches.
Patches allow developers to share and apply changes between different repositories or branches.

As part of a project I am working on, I found myself relying on this functionality and have explored a couple features. In this post, we will explore the world of Git patches, discussing their benefits and demonstrating how to utilize them effectively. We will cover relevant commands, provide code examples, and delve into the advantages of using rejection (rej) files and hunks.

What are Git Patches?
Creating Patches
- git format-patch
- Example: Creating a Patch
Applying Patches
- git apply
- Example: Applying a Patch
Using Rej Files
- Understanding Rej Files
- Resolving Conflicts with Rej Files
Working with Hunks
- Understanding Hunks
- Manipulating Hunks
Wrap up

1. What are Git Patches?

Git patches are a way to capture and transfer changes made to a codebase. A patch file contains a textual representation of the differences between two states of a repository. It includes information about added, modified, or deleted lines in the code.

Patches are valuable for sharing changes with others or applying them to different branches or repositories. They enable efficient code collaboration, particularly when dealing with remote teams or contributors.

2. Creating Patches

Git provides the git format-patch command to generate patches from commits or branches. Let's see how it works. I've made a repo as demo for this post which has the following commits.

`git format-patch` Command

The git format-patch command creates patch files for each commit in a specified range or branch. It generates files in the ".patch" format, which can be easily shared and applied.

Example: Creating a Patch

To create a patch for a specific commit, use the following command:



git format-patch <commit>

For instance, to generate a patch for the last three commits, execute:



git format-patch HEAD~2

This command will produce two patch files, each representing a commit.

3. Applying Patches

Once patches are created, they can be applied to a repository using the git apply command. Applying patches allows for seamless integration of changes into a codebase.

`git apply` Command

The git apply command is used to apply patches to a repository. It takes patch files and modifies the code accordingly.

Example: Applying a Patch

To apply a patch file, use the following command:



git apply <patch-file>

For example, to apply a patch named "this-is-my-patch.patch," run:



git apply this-is-my-patch.patch

The changes described in the patch file will be applied to the working directory.

A patch file contains something similar to a diff of the file and where the changes have occurred. The following is a sample of one of the patch files used for this demo.

4. Using Rej Files

If there's a mismatch between the file that you want to apply the patch to and the patch files, you will see a message similar to this.

To over come this, we can utilize rej files.

Rej files (rejection files) are generated during the patch application process when conflicts occur. They provide valuable information about rejected changes and aid in resolving conflicts effectively.

These are produced when the git apply command is used in a similar fashion:



git apply --reject --whitespace=fix this-is-my-patch.patch

Understanding Rej Files

When Git encounters conflicts while applying a patch, it creates a rej file for each conflicting hunk. A rej file highlights the conflicting lines and provides context for resolving the conflicts.

Resolving Conflicts with Rej Files

To resolve conflicts using rej files, follow these steps:

Open the rej file in a text editor.
Identify the conflicting lines and understand the context.
Manually modify the code to resolve the conflicts.
Use the git apply --reject command to reapply the patch, taking into account the manual modifications made.

Using rej files streamlines the conflict resolution process, ensuring accurate application of changes.

5. Working with Hunks

Hunks are portions of a patch file that describe a specific set of changes. Understanding and manipulating hunks allow for fine-grained control over patch application.

Understanding Hunks

Hunks represent the isolated changes within a patch file. They contain information about the lines affected, whether they were added, modified, or deleted, and context lines surrounding the changes.

Manipulating Hunks

To manipulate hunks during patch application, consider the following:

To apply only specific hunks from a patch file, use the --include or --exclude options with the git apply command.
To interactively choose individual hunks to apply, utilize the git add --patch command.

Manipulating hunks provides flexibility in applying changes, allowing for precise control over the patches.

6. Wrap up

Git patches are an indispensable tool for code collaboration and efficient transfer of changes between repositories. By understanding how to create and apply patches, utilize rej files, and work with hunks, developers can enhance their collaboration and streamline the integration of code.

Leveraging Git patches allows developers to easily share their changes, collaborate effectively with remote teams, and ensure seamless integration of code across branches and repositories. Understanding the commands and techniques associated with Git patches empowers developers to optimize their workflow and enhance productivity.

GL HF

Colab + Postgres + ngrok: A whopping trio!

PGzlan — Sun, 10 Sep 2023 20:49:53 +0000

In the previous post, we saw how to run Postgres on Google Colab and interface with it through psql as well as from Python itself. This only allowed accessing the database from within the Colab notebook - there was no way to connect from other applications or devices on the internet.

In this post, we will explore how to publicly expose the Postgres instance using ngrok, allowing secure connections from anywhere.

Wait. What is Ngrok?

Ngrock is a tunneling tool that allows you to expose a web service running on your local machine to the public internet over HTTPS. Here are some key things to know about ngrok:

It creates a secure tunnel between a public URL and a port on your local machine. This lets you share your locally-running services (like web apps) with others.
When you run ngrok, it provides you with a public URL that maps to your local server. For example, it might create a URL like https://abcdef.ngrok.io that tunnels to localhost:8080.
The URLs it generates are random subdomains under the ngrok domain, and change each time you restart ngrok. This helps protect your local server from being directly exposed.
It supports TCP and HTTP(S) tunnels. So you can expose web servers, databases, APIs and more using any protocol.
The tunnel is secure - it uses HTTPS and has built-in authentication. Ngrok handles encrypting and decrypting traffic between endpoints automatically.

So we can use ngrok to accessing our database from outside the local network, and exposing APIs for testing mobile or web clients or whoever we want.

Setting up ngrok

To start, we'll install ngrok on Colab:

You can grab your authtoken from the ngrok portal. To verify, you can open your Agents tab which is under Tunnels and you can the session that was started.

Connecting from another machine

Now on another (our local) computer, we can use the ngrok URL instead of localhost to reach our database.

We pass the ngrok URL along with username, password and database. The SSL connection is required due to ngrok's encryption.

You should now be connected to the Colab Postgres instance from your local machine! We can run queries and interact with the database remotely. Now your Python application has full access to the database running on Colab. The ngrok tunnel handles secure communication behind the scenes.

Other use cases

Exposing the database publicly opens up many possibilities beyond local experimentation:

Build test applications and connect from your laptop
Demo prototypes to clients without sharing Colab link
Integrate Colab work with larger apps via the database
Perform multi-user testing of database-backed services
Connect a web app to Postgres on Colab for development

Ngrok handles authentication securely, so you don't have to publicly expose credentials or ports. It's a very convenient tool for bringing services outside Colab.

Additionally, it might be possible to save your database on you google drive and then import your drive to continue between sessions.

Conclusion

In this post we saw how to use ngrok to forward port 5432 and make the Postgres instance on Google Colab publicly accessible via a web URL. This lets us connect and query the database from other machines, scripts, and applications securely.

Combined with Postgres and Ngrok, Colab provides a powerful free environment for developing, testing and prototyping full-stack database applications right from the browser. Ngrok opens new possibilities beyond the single notebook model, breaking Colab out of isolation when needed.

GL, HF.

Leveraging Google Colab to run Postgres: A Comprehensive Guide

PGzlan — Sun, 10 Sep 2023 16:54:03 +0000

Google Colab is a powerful cloud-based platform that provides a convenient environment for running Python code, performing data analysis, and executing machine learning tasks. While Colab offers built-in support for popular data manipulation libraries, such as Pandas and NumPy, connecting to a PostgreSQL database requires additional setup. In this guide, we'll explore how you can use Google Colab with Postgres, enabling you to leverage the power of SQL database operations within your Colab notebooks.

Prerequisites

Before we dive into the process, ensure you have the following prerequisites in place:

A Google account: You'll need a Google account to access Google Colab.
A PostgreSQL database: Ensure you have a working PostgreSQL database instance with the necessary credentials.

Setting up Google Colab

Open your web browser and navigate to Google Colab.
Sign in with your Google account if you haven't already.
Click on "New Notebook" to create a new Colab notebook.

Installing Required Libraries

Before doing anything, we need to check that the Postgres version we want is installed on Colab and then we can proceed, for the purposes of this walkthrough, we'll use Postgres 15



!sudo apt update
!sudo apt install dirmngr ca-certificates software-properties-common gnupg gnupg2 apt-transport-https curl -y
!curl -fSsL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor | sudo tee /usr/share/keyrings/postgresql.gpg > /dev/null
!echo 'deb [arch=amd64,arm64,ppc64el signed-by=/usr/share/keyrings/postgresql.gpg] http://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main' | sudo tee /etc/apt/sources.list.d/pgdg.list
!sudo apt update
!sudo apt install postgresql-client-15 postgresql-15 -y
!sudo service postgresql start

The following steps should end with the a screen similar to this

Once Postgres is in place, we should configure our users with proper permissions so that we can easily access our database. Let's create the following users:



# Setup a password `postgres` for username `postgres`
!sudo -u postgres psql -U postgres -c "CREATE ROLE root WITH SUPERUSER;"
!sudo -u postgres psql -U postgres -c "ALTER ROLE root WITH LOGIN;"
!sudo -u postgres psql -U postgres -c "CREATE ROLE postgres WITH PASSWORD 'postgres';"

Let's break down each line of the snippet:

!sudo -u postgres psql -U postgres -c "CREATE ROLE root WITH SUPERUSER;": This line executes a PostgreSQL command to create a new role (user) named "root" with superuser privileges. Superuser privileges provide extensive permissions and control over the database system. By creating this role, the snippet aims to establish a powerful user for administrative purposes.
!sudo -u postgres psql -U postgres -c "ALTER ROLE root WITH LOGIN;": This line executes a PostgreSQL command to allow the "root" role to log in. The WITH LOGIN clause grants the role the ability to authenticate and access the database system.
!sudo -u postgres psql -U postgres -c "CREATE ROLE postgres WITH PASSWORD 'postgres';": This line creates a new role named "postgres" with the password "postgres". The role is created with default privileges, which are typically limited compared to a superuser role. This setup allows for a more "secure" configuration by setting a password for the "postgres" role.

In my experience, setting root with LOGIN from the get-go usually fails for some reason. It also happens with the postgres user. If you encounter that, just run:



!sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'postgres'"

Now that's out of the way, let's see how we can use Postgres.

One of the issues that might seem annoying with Colab, is that if you run a cell for some jobs, it'll block the entire notebook from executing any cells. Luckily, we aren't trapped in all cases. There's an IPython extention called colabxterm that can be used to provide a terminal-like interface from a cell within the notebook. We can use this to out advantage. To install it, we just do the following



!pip install colab-xterm
load_ext colabxterm

After this is done, you can run



%xterm

To spawn a console

We can use this to connect to psql and run our queries!

To connect to a PostgreSQL database from Colab, we need to install the psycopg2 library, which allows Python to interact with PostgreSQL databases. Additionally, we'll install the sqlalchemy library, which provides a higher-level interface for working with databases.

To install the required libraries, add the following code snippet in a Colab cell:



!pip install psycopg2-binary sqlalchemy

Importing Necessary Modules

In the first cell of your Colab notebook, import the required modules:



import psycopg2
from sqlalchemy import create_engine

Connecting to the PostgreSQL Database

To establish a connection to your PostgreSQL database, you need to provide the necessary connection details, such as the host, port, database name, username, and password.

Use the following code snippet as an example to establish a connection using psycopg2:



try:
    connection = psycopg2.connect(
        host="your_host",
        port="your_port",
        database="your_database",
        user="your_username",
        password="your_password"
    )
    cursor = connection.cursor()
    print("Connection established successfully!")
except Exception as e:
    print("Error connecting to the database:", e)

Replace the placeholders (your_host, your_port, your_database, your_username, your_password) with your actual database connection details.

Alternatively, you can use the sqlalchemy library to create an engine and connect to the database:



try:
    engine = create_engine('postgresql://your_username:your_password@your_host:your_port/your_database')
    connection = engine.connect()
    print("Connection established successfully!")
except Exception as e:
    print("Error connecting to the database:", e)

Replace the placeholders (your_host, your_port, your_database, your_username, your_password) with your actual database connection details.

Executing SQL Queries

Once the connection is established, you can execute SQL queries and fetch the results using the cursor object (if using psycopg2) or the execute method of the engine (if using sqlalchemy).

Here's an example that executes a simple SELECT query and fetches the results using psycopg2:



try:
    cursor.execute("SELECT * FROM your_table;")
    results = cursor.fetchall()

    for row in results:
        print(row)
except Exception as e:
    print("Error executing the query:", e)

Replace your_table with the name of the table you want to query from your database.

For sqlalchemy, you can execute queries using the execute method of the engine object:



try:
    result = connection.execute("SELECT * FROM your_table")
    for row in result:
        print(row)
except Exception as e:
    print("Error executing the query:", e)

Replace your_table with the name of the table you want to query from your database.

Closing the Connection

After you finish executing the queries, remember to close the connection to the database to release any resources.

Use the following code to close the connection when using psycopg2:



if connection:
    cursor.close()
    connection.close()
    print("Connection closed successfully!")

For sqlalchemy, close the connection as follows:



if connection:
    connection.close()
    print("Connection closed successfully!")

Here's what this would look like:

Conclusion

In this guide, we explored how to use Google Colab with a PostgreSQL database. By following the steps outlined here, you can establish a connection to your database, execute SQL queries, and retrieve the results within your Colab notebooks. This integration allows you to leverage the power of both Colab's data analysis capabilities and the SQL operations provided by Postgres, enabling you toperform advanced data manipulations and gain valuable insights.

One of the key advantages of using Google Colab with Postgres is the ability to seamlessly combine the data analysis capabilities of Colab with the power of SQL queries. With Colab, you have access to a wide range of Python libraries for data manipulation and visualization. By connecting to a Postgres database, you can leverage the SQL language to efficiently retrieve and manipulate data directly from your database.

For example, you can use SQL queries to perform aggregations, filtering, and join operations on large datasets stored in your Postgres database. These operations can be executed efficiently on the database server, reducing the data transfer overhead and improving performance. Once you have the desired data subset, you can use Colab's data analysis libraries, such as Pandas and Matplotlib, to further explore and visualize the data.

Integrating Google Colab with Postgres opens up a world of possibilities for data analysis and exploration. By combining the strengths of Colab's Python libraries and Postgres' SQL capabilities, you can efficiently work with large datasets, perform advanced data manipulations, and gain valuable insights. So, give it a try and unlock the full potential of your data analysis workflows with Google Colab and Postgres!

GL, HF!

PyScript: Python in the Browser

PGzlan — Sat, 12 Aug 2023 18:18:59 +0000

PyScript is tool that allows Python code to run directly in the browser via web assembly (Wasm).

This opens up opportunities to use Python for frontend web development and interact with JavaScript environments from Python. In this blog post, we'll explore what PyScript is, how to use it, and provide some code examples.

What is PyScript?

PyScript is a framework that allows users to create rich Python applications in the browser using HTML’s interface and the power of Pyodide, WASM, and modern web technologies. The PyScript framework provides users at every experience level with access to an expressive, easy-to-learn programming language with countless applications. [It enables users to run Python code directly in their browser, giving anyone the ability to program without infrastructure barriers. You can add an interactive Python REPL directly to your website, share an interactive dashboard with a colleague as an HTML file, or create a client-side Python-powered web application.

Some of the key features of PyScript include:

It allows users to write and run Python code in HTML.
It allows users to build web applications using Python
It allows sharing machine learning models in an HTML file so that others can run those models in their browsers
It helps to avoid setup
It allows using UI components such as buttons, containers, text boxes, etc

So in summary, PyScript allows writing frontend code with the Python programming language and interacting seamlessly with JavaScript environments, all running natively in the browser via WebAssembly.

How do I use PyScript?

To use PyScript, you do not require any development environment other than a web browser (Chrome is recommended. Long Live 🔥🦊) and a text editor, even though using your IDE of choice might be convenient

That's it. There is no installation required. The PyScript assets served on pyscript.net which makes the process of getting up and running pretty easy.

Our first PyScript file

Using your favorite editor, create a new file called with the html extension (This ideally should be in the same directory as your support Python Script(s), JavaScript, and CSS files with the following content, and open the file in your web browser.

For the sake of brevity, let's assume the file you created is called index.html

<html>
  <head>
    <link rel="stylesheet" href="https://pyscript.net/latest/pyscript.css"/>
    <script defer src="https://pyscript.net/latest/pyscript.js"></script>
  </head>
  <body>
    <input type="color" id="color-picker">
    <div id="text" style="font-size: 48px">Hello World!</div>
    <py-script>
from js import document
from pyodide.ffi import create_proxy
color_picker = document.getElementById("color-picker")
text = document.getElementById("text")
def on_color_change(event):
    text.style.color = event.target.value
color_picker.addEventListener("change", create_proxy(on_color_change))
    </py-script>
  </body>
</html>

We start by linking PyScript and you you can do that in two ways:

Download a copy of the latest version of PyScript, and add the required files to head in your HTML document:
Add the CDN version of PyScript's JS and CSS files to head in your HTML document:

This script is an HTML file that uses PyScript to create a color picker and change the color of the text based on the selected color. The logic is present inside the <py-script> tag

<html>
  <head>
      ...
  </head>
  <body>
    <input type="color" id="color-picker">
    <div id="text" style="font-size: 48px">Hello World!</div>
    <py-script>
from js import document
from pyodide.ffi import create_proxy

color_picker = document.getElementById("color-picker")
text = document.getElementById("text")

def on_color_change(event):
    text.style.color = event.target.value

color_picker.addEventListener("change", create_proxy(on_color_change))
    </py-script>
  </body>
</html>

When you open this file in a web browser, you will see an input element of type color that allows you to select a color, and a div element containing the text “Hello World!” in a large font size.

The script uses PyScript to select the color picker and the div element using their IDs, and defines a function on_color_change that is called whenever the value of the color picker changes.

In this function, the style.color property of the div element is set to the value of the color picker.

The script also attaches an event listener to the color picker to call the on_color_change function whenever its value changes. As a result, when you select a color using the color picker, the color of the text will change accordingly.

If you take a closer look at the addEventListener, you'll notice a function called create_proxy that wraps the on_color_change function.

This purpose of this function comes from the fact that Pyodide needs to briefly proxy the Python function on_color_change so the JS function addEventListener knows how to interact with it. But once addEventListener terminates, that proxy is no longer needed, it gets destroyed... and then when an event comes around to trigger your function, the proxy it should be using is gone. This will cause the script to return the following error

Uncaught Error: This borrowed proxy was automatically destroyed at the end of a function call. Try using create_proxy or create_once_callable.
The object was of type "function" and had repr

The two functions that are mentioned in the error create a PyProxy (a JS object) of your Python object that you, the user, are supposed to manage the lifetime of, by calling PyProxy.destroy() on it when you're done with it. Or, if you use create_once_callable(), the proxy will destroy() itself after the first time it's called.

For an object such as an event listener, destroying the proxy for the lifetime of your page may not be a desirable thing, so you can just leave it hanging around. It's worth noting that if you remove that event listener or the element attached to it, you should plan to track and destroy the PyProxy, to avoid taking up memory.

Interacting with JavaScript

A major benefit of PyScript is being able to directly interact with JavaScript from Python code. This enables full integration with frontend frameworks and libraries.

PyScript provides a js module that exposes JavaScript functions and objects to Python. For example:

<html>
  <head>
    <link rel="stylesheet" href="https://pyscript.net/latest/pyscript.css"/>
    <script defer src="https://pyscript.net/latest/pyscript.js"></script>
    <script>
    name = "🦊";
    function sayMyName(name){
        console.log(name);
    }
    </script>
  </head>
  <body>
    <py-script>
from js import name, sayMyName

print(f"Hello {name}")
print("Saying your name in Javascript: " + str(sayMyName('Dev.to')))
    </py-script>
  </body>
</html>

Here we define a Python function that calls out to JavaScript's console.log(). When compiled and run, it will print the passed name. It can also be seen in the console of the web browser.

We can also access global JavaScript objects. For example, to display a alert popup:

import js

js.alert("Hello from Python!")

So PyScript provides full interoperability between Python and JavaScript for building hybrid applications.

Async/Await Support

Another key feature of PyScript is built-in support for asyncio and async/await syntax. This enables writing asynchronous Python code that integrates cleanly with async-based JavaScript code and event loops.

For example, we can write an async function that performs a simulated async task:

<html>
  <head>
    <link rel="stylesheet" href="https://pyscript.net/latest/pyscript.css"/>
    <script defer src="https://pyscript.net/latest/pyscript.js"></script>
  </head>
  <body>
    <py-script>
import asyncio
import js
from pyodide.ffi import create_proxy

async def do_task():
  await asyncio.sleep(1)
  print("Task finished!")

async def main():
  await do_task()
  print("Done!")

js.setTimeout(create_proxy(main), 0)  
    </py-script>
  </body>
</html>

Here we define an async function do_task() that sleeps for 1 second, and call it from the main function main(). We schedule main() to run asynchronously after 0 seconds using JavaScript's setTimeout().

When compiled and run, this will output the lines with a 1 second delay in between, demonstrating asyncio integration with the browser's event loop.

Conclusion

PyScript provides an intriguing new way to use Python for frontend web development by compiling it to WebAssembly. It maintains the core Python syntax and semantics while enabling seamless interoperability with JavaScript.

Some key advantages it offers include:

Familiar Python syntax for writing frontend code
Full asyncio/await support for asynchronous programming
Interactive usage directly in browsers via debugging tools
Performance comparable to JavaScript with WebAssembly compilation
Reuse of Python skills, libraries and frameworks

PyScript is a powerful tool for building frontend apps with Python. I'm excited to see where the project goes and do intend to play around with it a little more

Google Colab: Boost your Notebook Experience

PGzlan — Sat, 12 Aug 2023 14:19:15 +0000

Google Colab, or Colaboratory, is a free online coding environment that offers a platform for anyone who wants to develop machine learning applications. It's fast, easy to use, and best of all, it gives you access to Google's hardware, including GPUs and TPUs, absolutely free of charge! .

What is Google Colab?

Google Colab is based on Jupyter Notebook, a popular open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text. Colab extends the Jupyter Notebook system, enabling you to write and execute Python code (in lucky instances R) in your browser, with zero configuration required, free access to GPUs, easy sharing, and much more.

Getting started with Google Colab

Starting with Google Colab is as simple as logging into your Google account and then accessing the Colab at colab.research.google.com.

Once you're there, you can create a new notebook by clicking on File -> New notebook. A new tab will open with your fresh notebook.

The interface is divided into two main areas:

The main toolbar which has options to share, save, connect to GitHub, etc.
The cell space where you can write your code or text.

Each notebook is made up of cells. There are two types of cells:

Code cells: Where you write and execute your code.
Text cells: Where you write text or HTML code for explanations, much like in this blog post.

You can add a new cell by clicking on + Code or + Text on the top left corner of your notebook.

How to use Google Colab

Let's dive into a simple example that demonstrates how to write, run code, and use the output.

First, add a new code cell and write the following Python code:

print("Hello, Google Colab!")

Click the play button on the left or press Shift+Enter to run the cell. The output will be displayed right below the cell:

Hello, Google Colab!

This is a simple Python code execution. Now, let's see how we can use Google Colab to run a more complex task, such as training a machine learning model. We'll use the popular Iris dataset from the scikit-learn library to train a logistic regression model.

# Importing necessary libraries
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# Load Iris dataset
iris = load_iris()
X = iris.data
y = iris.target

# Split the data for training and testing
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Define the model
model = LogisticRegression()

# Train the model
model.fit(X_train, y_train)

# Evaluate the model
score = model.score(X_test, y_test)

print(f'Model accuracy: {score*100:.2f}%')

This code first loads the Iris dataset, then splits it into a training set and a test set. A Logistic Regression model is then defined and trained, and finally, the model's accuracy is evaluated and printed.

Using Colab's GPU

One of the significant advantages of Google Colab is the free GPU that Google provides for your computations, which can be incredibly beneficial for tasks like training deep learning models. To use the GPU, you need to change your notebook settings by going to Runtime -> Change runtime type and selecting GPU from the Hardware accelerator dropdown menu.

Once you've done that, you can verify that you're using GPU by running the following code:

import tensorflow as tf

# Check if GPU is available
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  raise SystemError('GPU device not found')
print(f'Found GPU at: {device_name}')

If the GPU is correctly configured, the output will be something like:

Found GPU at: /device:GPU:0

In the current version of colab, you can see the GPU name in the top right corner which has like a small resource monitor

Without GPU	With GPU

Colab Forms

Colab Forms provide an easy way to parameterize code in Google Colab notebooks. From a code cell, you can select Insert → Add form field to add a form field. When you change the value in a form, the corresponding value in the code will change. Forms support many types of fields, including string fields, raw fields, date fields, number fields, boolean fields, and markdown. You can also hide the code associated with a form by selecting View → Show/hide code or using the toolbar above the selected code cell

String fields: You can use string fields to allow users to enter text. For example:

text = 'value' #@param {type:"string"}
dropdown = '1st option' #@param ["1st option", "2nd option", "3rd option"]
text_and_dropdown = 'value' #@param ["1st option", "2nd option", "3rd option"] {allow-input: true}

Number fields: You can use number fields to allow users to enter numbers. For example:

number_input = 10.0 #@param {type:"number"}
integer_input = 10 #@param {type:"integer"}

Sliders: You can use sliders to allow users to select a value from a range. For example:

number_slider = 0 #@param {type:"slider", min:-1, max:1, step:0.1}
integer_slider = 1 #@param {type:"slider", min:0, max:100, step:1}

Date fields: You can use date fields to allow users to select a date. For example:

date_input = '2018-03-22' #@param {type:"date"}

Boolean fields: You can use boolean fields to allow users to select a true or false value. For example:

boolean_checkbox = True #@param {type:"boolean"}
boolean_dropdown = True #@param ["False", "True"] {type:"raw"}

Raw fields: You can use raw fields to allow users to enter any value without type checking. For example:

raw_input = None #@param {type:"raw"}
raw_dropdown = raw_input #@param [1, "raw_input", "False", "'string'"] {type:"raw"}

Colab Magics

Colab provides a set of commands that provide a mini command language for Google Colab notebooks. They are called "Magics", which are a feature of IPython, which is the kernel used by Google Colab. These commands are orthogonal to the syntax of Python and are extensible by the user with new commands. Magics come in two kinds: Line magics, which are commands prepended by one % character and whose arguments only extend to the end of the current line, and Cell magics, which use two percent characters as a marker (%%), and they receive as argument both the current line where they are declared and the whole body of the cell.

You can list all available magic commands by using the %lsmagic command. This command will display a list of all the available line and cell magics. You can also access the help module for magic commands by using the %magic command. You can also bring up the help module for a specific command by appending a ? to the end of the command, for example %timeit?

You can create your own custom magic commands in Google Colab. Magic commands are a feature of IPython, which is the kernel used by Google Colab. IPython has a system of commands called ‘magics’ that provide a mini command language that is orthogonal to the syntax of Python and is extensible by the user with new commands. Magics come in two kinds: Line magics, which are commands prepended by one % character and whose arguments only extend to the end of the current line, and Cell magics, which use two percent characters as a marker (%%), and they receive as argument both the current line where they are declared and the whole body of the cell

from IPython.core.magic import register_line_magic

@register_line_magic
def greet(line):
    name = line.strip()
    print(f"Hello, {name}!")

# Register the magic command
get_ipython().register_magic_function(greet, magic_kind='line')

You can also use line magic within your code to do some actions such as changing directory

Run system commands

You can run system commands in Google Colab by using the ! character before the command.

Since the underlying host of this notebook is Linux, you can run all the Linux commands you desire.

For example, to run the ls command, you would enter !ls in a code cell and run it. This will execute the ls command and display the output in the cell. You can also use this method to run more complex commands and even chain multiple commands together using &&. For example, to update the package list and then install a package, you could enter !apt-get update && apt-get install -y package-name in a code cell and run it.

This can be useful for cases when you want to perhaps do some processing on some video (perhaps extract a part of it based on timestamps) with a certain tool such as ffmpeg, you can run your ffmpeg command before executing your code

Conclusion

Google Colab is a powerful tool for anyone who wants to dive into machine learning without worrying about setting up an environment or the computational resources (you can also use it to test out things quickly if especially if you're testing something Linux related). It has an easy to get into user interface for coding and collaboration, and the free access to Google's hardware makes it a go-to choice for many data scientists and researchers.

So, start integrating Google Colab into your workflow to take it to the next level

Note: While Google Colab is a fantastic resource, remember that it is not meant for long, continuous usage. The session will disconnect after a period of inactivity. You can check around for how you can reduce the likelyhood of that happening

DEV Community: PGzlan

Identity and Access Management: Explained

Identity Management (IdM)

Identity and Access Management (IAM)

Discretionary Access Control

Role-Based Access Control

Mandatory Access Control

Attribute-Based Access Control

Python's IceCream Package: Effortless Debugging with Minimal Code

Introduction

Usage

Multiple Values

Conditional Statements

Iterables

Functions

Nifty things about ic

Conclusion

References

Python Walrus Operator: An Introduction

What is the Walrus Operator?

Basic Usage

Use Cases

1. If Statements

2. For Loops

3. List Comprehension

4. While Loops

Controversy Surrounding the Walrus Operator

Conclusion

Resources

Exploring Different Types of Joins in Pandas

Understanding Joins

Inner Join

Left Join

Right Join

Outer Join

Conclusion

References

MITRE ATT&CK: Your map to adversary movement

Table of Contents

1. Introduction to MITRE ATT&CK

2. The Framework

2.1. Tactics

2.2. Techniques

2.3. Sub-techniques

2.4. Procedures

3. How MITRE ATT&CK Works

3.1. Adversary Emulation

3.2. Threat Intelligence

4. The Benefits of MITRE ATT&CK

4.1. Improved Threat Detection

4.2. Enhanced Incident Response

4.3. Better Security Assessments

5. Implementing MITRE ATT&CK

5.1. Mapping Techniques and Controls

5.2. Leveraging MITRE ATT&CK for Red Teaming

6. Limitations and Challenges

7. Conclusion

Using Shared Memory in Linux

Introduction:

Table of Contents

1. Introduction to Shared Memory

2. Shared Memory Functions

shm_open

shm_unlink

shmget and shmat

shmdt

3. Benefits of Using Shared Memory

4. Security Risks: Data Read Off Disk

Understanding the Vulnerability

Mitigating Risks

5. Code Example

6. Conclusion

References

Efficiency and Speed with GNU Parallel

Table of Contents

Introduction

1. Understanding GNU Parallel

2. Installation and Basic Usage

3. Parallelization Modes and Job Control

4. Advanced Features and Examples

Nifty things about `ic`

`shm_open`

`shm_unlink`

`shmget` and `shmat`

`shmdt`

`git format-patch` Command

`git apply` Command