DEV Community: Felix

Introduction to EC2 instance metadata and risk prevention

Felix — Sun, 09 Jul 2023 15:40:47 +0000

Introduction

Amazon EC2 Instance Metadata Service (IMDS) can help users obtain information about the instance itself. With IMDS, you can access various information about the instance, such as hostname, host IP, temporary access credentials, user data, and more. While this information is convenient for users, it also introduces new risks.

Risks

A common attack scenario is when an instance has an SSRF vulnerability, attackers can retrieve information from the metadata. If temporary access credentials exist in the metadata, attackers can use these credentials for lateral attacks.

!https://superblog.supercdn.cloud/site_cuid_cldhcpt7a1506291jkdhe6vl2sc/images/untitled-11-1688916597258-compressed.png

Furthermore, if sensitive information is present in user data, attackers can retrieve this data through the metadata, resulting in information leakage and facilitating the next steps of attack and exploitation by the attackers.

Risk Detection

Manual Assessment

There are three main ways to assess the risks:

Access http://169.254.169.254/latest/meta-data/ within the instance to check if the returned result is normal. If it is, it indicates that the instance's metadata is enabled and accessible without authorization.
In the instance's Action → Instance settings → Modify instance metadata options, if the Instance metadata service is in an "Enable" state and IMDSv2 is in an "Optional" state, it means the metadata is at risk.
You can also use the AWS CLI to obtain the current instance's state:

aws ec2 describe-instances --instance-ids <instance_id> --query 'Reservations[*].Instances[*].MetadataOptions'

If the HttpTokens in the returned content is "optional," it indicates that the metadata service of the current instance is at risk.

[
    [
        {
            "State": "applied",
            "HttpTokens": "optional",
            "HttpPutResponseHopLimit": 1,
            "HttpEndpoint": "enabled",
            "HttpProtocolIpv6": "disabled",
            "InstanceMetadataTagsF!": "disabled"
        }
    ]
]

Using Selefra for Assessment

Manual discovery of these issues can be time-consuming and cannot be done in bulk. Using Selefra can help you quickly identify these risks. Selefra is a tool for rapidly discovering multi-cloud and SaaS risks. The Selefra project can be found at github.com/selefra/selefra.

Let's start by installing Selefra:

brew tap selefra/tap
brew install selefra/tap/selefra

Next, create a project folder:

mkdir selefra-test
cd selefra-test

Copy the following YAML file to this folder:

selefra:
    name: selefra-test
    connection:
      type: postgres
      username: your_username
      password: your_password
      host: 127.0.0.1
      port: 5432
      database: postgres
      sslmode: disable
    log_level: info
    providers:
        - name: aws
          source: aws
          version: v0.1.0
providers:
    - name: aws
      provider: aws
      cache: 7d
rules:
  - name: ec2_metadata_unlimited_access
    metadata:
      title: EC2 metadata unlimited access
      severity: High
    query: |-
      SELECT
                *
            FROM
                aws_ec2_instances
            WHERE
                metadata_options ->> 'HttpTokens' = 'optional';
    output: "EC2 metadata unlimited access, arn: { {.arn} }"

As you can see, it is divided into three modules: selefra, provider, and rules.

In the selefra module, configure your own PostgreSQL database connection address, username, and password in the connection block. The cache block in the provider module can set the cache time for fetching data. The rules module is for configuring the detection issues. The title block represents the title of the detection strategy, and the SQL statement in the query block is used to execute the detection strategy by querying the vulnerable resources in the database.

Before starting the detection, configure your AWS credentials by running the following command:

aws configure

Then run the following command to run Selefra:

selefra apply

Now Selefra will start the detection process. Below is an example of the result:

In the result, we can see that there are 3 instances with metadata risks.

In addition to the above method, Selefra also integrates GPT functionality. This feature allows you to discover risk points by directly querying Selefra.

Using Selefra GPT for Assessment

Similar to the previous steps, create a new folder and copy the following YAML file into it:

selefra:
    name: selefra-test
    connection:
      type: postgres
      username: yourusername
      password: yourpassword
      host: 127.0.0.1
      port: 5432
      database: postgres
      sslmode: disable
    log_level: info
        openai_api_key: your_openai_api_key
    openai_mode: gpt-4
    openai_limit: 10
    providers:
        - name: aws
          source: aws
          version: v0.1.0
providers:
    - name: aws
      provider: aws
      cache: 30d
rules:

The difference here is that you need to provide your own OpenAI API Key. You can also set whether to use GPT-4 or GPT-3.5 in the openai_mode. Furthermore, leave the rules block empty, as AI will generate the content for this section.

Before starting the detection, configure your AWS credentials as before. Then you can use the GPT functionality:

selefra gpt "Query instances with unrestricted access to metadata."

As you can see, with just a single sentence, you can discover instances where the metadata service is at risk. It's very convenient.

Prevention

To mitigate the risks associated with the AWS EC2 Instance Metadata Service, there are two main methods. First, if you don't need metadata, you can disable it directly. Second, if you need to use the metadata service, you can enable token-based access to prevent direct access to metadata.

To disable the metadata service, there are three common methods:

1.Disable metadata during instance creation.

2.After creating the instance, go to the console's Action → Instance settings → Modify instance metadata options and disable it.

After creating the instance, you can also use AWS CLI to disable it:

aws ec2 modify-instance-metadata-options --instance-id <instance_id> --http-endpoint disabled

To enable token-based access, there are also three common methods:

1.During instance creation, select "V2 only" in the Metadata version.

2.After creating the instance, go to the console's Action → Instance settings → Modify instance metadata options, and check "Required" for IMDSv2.

After creating the instance, you can also use the command line to modify it:

aws ec2 modify-instance-metadata-options --instance-id <instance_id> --http-tokens required

Conclusion

In this article, we introduced the risks associated with metadata and how to mitigate these risks. By using Selefra, you can quickly and efficiently identify instances that have metadata risks. We hope this article has been helpful to you and that Selefra can make your cloud environment more secure.

GitHub: https://github.com/selefra/selefra

Slack: https://selefra.io/community/join

Recommended reading

How to spot and fix issues with publicly accessible AWS S3 buckets

Why Choose PGSQL as the Database for Resource Storage

Felix — Sun, 09 Jul 2023 10:38:19 +0000

Analyzing the Ideal Database Choice for Selefra's Policy-as-Code Tool

Introduction

In today's cloud computing environment, the demand for Policy-as-Code (PaC) tools is increasing to achieve tasks such as multi-cloud security, cost analysis, and organizational structure analysis. Selefra is an excellent PaC tool that uses YAML to read policies and relies on PGSQL database for storing cloud service resources. This article will explore why PGSQL is chosen as the database for resource storage and highlight its advantages for such query analysis PaC tools.

Scalability

PGSQL is a mature open-source Relational Database Management System (RDBMS) with excellent scalability. In the cloud environment, the number and scale of resources can grow over time. PGSQL's scalability allows it to handle large volumes of data and supports horizontal and vertical scaling, meeting the storage requirements of Selefra in different-scale environments.

Data Consistency

As a relational database, PGSQL provides robust transaction support, ensuring data consistency. Selefra needs to read and analyze data from cloud service resources and perform policy checks based on it. PGSQL's transaction mechanism guarantees the integrity and consistency of data across multiple queries and analysis operations, enhancing the tool's reliability and accuracy.

Powerful Query Capabilities

Selefra requires flexible and efficient querying and analysis of cloud service resources. PGSQL offers rich query capabilities, supporting complex SQL queries, aggregation operations, and window functions, among others. By leveraging PGSQL's powerful querying capabilities, Selefra can easily perform in-depth analysis of resources in multi-cloud environments, providing valuable insights to users.

Such as:

SELECT
  DISTINCT(a1.*)
FROM
  aws_s3_buckets a1,
  aws_s3_bucket_grants a2
WHERE
 a1.selefra_id = a2.aws_s3_buckets_selefra_id
  AND a2.grantee :: jsonb ->> 'URI' = 'http://acs.amazonaws.com/groups/global/AllUsers'
  AND a2.permission IN ('WRITE_ACP', 'FULL_CONTROL');

Security

In PaC tools, security is a crucial concern, especially when dealing with sensitive data. PGSQL provides multi-level security features, including role-based access control, SSL/TLS encryption, and transparent encryption storage of data. By using PGSQL as the database for resource storage, Selefra can leverage these security features to protect user data, ensuring that sensitive information remains inaccessible to unauthorized parties.

Mature Ecosystem

PGSQL, as a popular database management system, has a large and active community and ecosystem. This means that support and advice on performance tuning, security best practices, and bug fixes are readily available. Selefra can benefit from the knowledge and resources of the PGSQL community, staying up-to-date with the latest developments in database technology and obtaining support when needed.

Conclusion

Choosing PGSQL as the database for resource storage in Selefra is a wise decision. PGSQL's scalability, data consistency, powerful query capabilities, security features, and mature ecosystem make it an ideal choice for query analysis-based Policy-as-Code tools. By harnessing the advantages of PGSQL, Selefra can provide efficient, reliable, and secure analysis of cloud service resources, offering users a better experience in multi-cloud management.

GitHub: https://github.com/selefra/selefra

Slack: https://selefra.io/community/join

A Better Version Is Released - Selefra v0.2.3

Felix — Tue, 04 Jul 2023 04:17:25 +0000

Release Note - Selefra v0.2.3

Date: June 30, 2023

We are excited to announce the release of Selefra v0.2.3, an open-source Policy-as-Code (PaC) product. This version introduces several bug fixes and enhancements to improve your experience with Selefra.

Here are some key updates:

[Feature]Modules support filtering, while labels support customization of any format. by @FelixsJiang in #30
[Fixed]Change the uses to string. by @FelixsJiang in #30
[Feature]Add output error handling. by @FelixsJiang in #30

In the new version, you can filter out unnecessary policies by configuring a filter in modules to streamline the results.

Specifically, you can configure it as follows：

modules:
  - name: aws-misconfiguration-s3
    uses: ./rules
    filter:
      - name: ebs_encryption_is_disabled_by_default
        severity: Low

This example will filter out the policy with name is ebs_encryption_is_disabled_by_default or the Policy with low severity.

Summarize

To get started with Selefra v0.2.3, please visit our official repository (Github: https://github.com/selefra/selefra) and follow the installation instructions. Remember to update your current version to benefit from the latest enhancements.

Thanks for your continued support, and we hope you find Selefra v0.2.3 to be a valuable PaC implementing tool in your projects.

Enjoy!

The Selefra Team

Github: https://github.com/selefra/selefra

How to spot and fix issues with publicly accessible AWS S3 buckets

Felix — Sun, 02 Jul 2023 08:35:21 +0000

Introduction

The issue of public access to AWS S3 buckets is one of the most common problems encountered with AWS S3 services. This can lead to sensitive data stored in S3 being exposed. The presence of this problem is due to the configuration of public access policies for the storage bucket. In this article, we will discuss how to detect and prevent this issue.

Understanding the Problem

When we can directly access files in a storage bucket, it indicates that the bucket has a public access issue.

As seen in the image above, we can directly access files in the storage bucket. Now let's understand why this is happening.

If we examine the policy for this bucket, we might see a configuration similar to the following:

{
    "Version": "2012-10-17",
    "Id": "test",
    "Statement": [
        {
            "Sid": "test",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::selefra-test-omkqt/*"
        }
    ]
}

In this policy, everyone is granted the s3:GetObject permission for the selefra-test-xxxx bucket, resulting in public access to objects in the bucket.

Remediation

We should follow the principle of least privilege, which means allowing only specific users to have specific permissions instead of granting access to all users.

For storage buckets that require public access due to business requirements, sensitive data should be avoided in those buckets.

Using Selefra for Quick Discovery

Manually discovering these issues can be time-consuming and cannot be done in bulk. Using Selefra can help you quickly identify these risks.

Selefra project repository: github.com/selefra/selefra

Typical Usage of Selefra

First, let's install Selefra:

brew tap selefra/tap
brew install selefra/tap/selefra

Next, create a new project folder:

mkdir selefra-test
cd selefra-test

Copy the following YAML file into this folder:

selefra:
    name: selefra-test
    connection:
      type: postgres
      username: your_username
      password: your_password
      host: 127.0.0.1
      port: 5432
      database: postgres
      sslmode: disable
    log_level: info
    providers:
        - name: aws
          source: aws
          version: v0.1.0
providers:
    - name: aws
      provider: aws
      cache: 7d
rules:
    - name: bucket_publicly_readable
        metadata:
      title: S3 bucket public readable
    query: |-
      SELECT
        DISTINCT(a1.*)
      FROM
        aws_s3_buckets a1,
        json_array_elements (a1.policy :: json -> 'Statement') a2
      WHERE
        (
          a2 ->> 'Action' = 's3:GetObject'
          OR a2 ->> 'Action' = 's3:Get*'
          OR a2 ->> 'Action' = 's3:*'
        )
        AND a2 ->> 'Effect' = 'Allow'
        AND (
          a2 ->> 'Principal' = '*'
          OR a2 -> 'Principal'

 ->> 'AWS' = '*'
        );
        output: "S3 bucket public readable, arn: {{.arn}}"

In the Selefra module, configure your own PostgreSQL database connection address, username, and password in the connection block. The cache block in the providers module sets the data retrieval cache time. The rules module is related to the configuration for detecting issues. The title block represents the title of the detection policy, and the SQL query block is used to execute the detection policy. It executes this SQL query in the database to search for resources with risks.

Before starting the detection, configure your AWS credentials using the following command:

aws configure

Then run the following command to start Selefra:

selefra apply

Selefra will start the detection process. Here is an example of the result:

In the result, we can see the at-risk buckets. Apart from the above method, Selefra also integrates a ChatGPT feature that allows you to discover risk points by directly asking Selefra.

Selefra GPT Feature

Similar to the previous steps, create a new folder and copy the following YAML file into the folder:

selefra:
    name: selefra-test
    connection:
      type: postgres
      username: yourusername
      password: yourpassword
      host: 127.0.0.1
      port: 5432
      database: postgres
      sslmode: disable
    log_level: info
        openai_api_key: your_openai_api_key
    openai_mode: gpt-4
    openai_limit: 10
    providers:
        - name: aws
          source: aws
          version: v0.1.0
providers:
    - name: aws
      provider: aws
      cache: 7d
rules:

In this case, you need to provide your OpenAI API key and choose whether to use GPT-4 or GPT-3.5 in the openai_mode field. Additionally, keep the rules block empty as the AI will generate it automatically.

Before starting the detection, configure your AWS credentials, and then you can use the GPT feature:

selefra gpt "Query publicly accessible S3 buckets."

As seen in the example above, with a simple sentence, you can find publicly accessible buckets. It's very convenient.

Conclusion

Public access to S3 buckets is a frequent and pressing issue. We hope this article has helped you understand and address the problem of public access to storage buckets in AWS S3. Selefra makes the cloud more secure.

How Selefra Combines GPT for Multi-Cloud Security Scanning

Felix — Fri, 30 Jun 2023 06:17:11 +0000

Introduction

When combining GPT with risk scanning, Selefra adopts an innovative approach by integrating the powerful capabilities of PolicyAsCode and GPT to enhance the efficiency and accuracy of multi-cloud security compliance detection. While PolicyAsCode has been widely used for internal team collaboration and external open-source project collaboration, as well as facilitating version tracking and audit tracing, it can present challenges due to its high learning curve and lengthy and complex code.

To overcome these challenges, Selefra introduces GPT (Generative Pre-trained Transformer) technology to reduce the complexity and provide a more professional risk scanning solution. GPT is a deep learning-based natural language processing model known for its strong semantic understanding and generation capabilities. By combining GPT with PolicyAsCode, Selefra enables an intelligent security compliance analysis workflow.

Within Selefra, specific GPT prompts are designed for security compliance, cost calculation, and architecture design. Users can trigger automated analysis by inputting GPT statements and select appropriate pre-defined prompts based on their inputs. This flexible mechanism ensures customized risk analysis tailored to different requirements.

Expert Simulation

In Selefra, we have designed dedicated GPT prompts for security compliance, cost calculation, and architecture design. By analyzing the user's input GPT statements, Selefra automatically determines which pre-defined prompt to utilize for the analysis. Let's briefly explain how GPT is used for security compliance project analysis.

Resource Acquisition

Before conducting the analysis with GPT, we still need to acquire resources through Selefra's provider, with some differences in the policy invocation process.

Resource Analysis

In the regular mode, policies typically consist of user-configured SQL statements that query the acquired database to identify risky data and assess the risk level of resources. However, when utilizing GPT, Selefra first submits all the tables to GPT for analysis to determine which tables require analysis. Then, based on the results returned by GPT, Selefra organizes the corresponding table's resource content and sequentially submits it to GPT for analysis. GPT compiles the analysis results into a predefined output format, which is rendered by Selefra to present the structured output to the user.

For the specific code implementation, you can refer to the following four methods in the /pkg/modules/executors/module_query_executor.go file: filterTables, filterColumns, getRows, and getIssue.

Summary

In summary, leveraging GPT for multi-cloud product risk scanning significantly reduces the complexity of using PolicyAsCode products and enhances the accuracy of the analysis. However, it is essential to note that the accuracy of the GPT analysis depends on the quality of its training data and models. Therefore, this approach is ideal for beginners and serves as an introductory method for risk scanning, while becoming proficient in risk scanning techniques still requires learning the specialized analysis syntax of PolicyAsCode.

Thank you for reading.

Selefra GitHub: https://github.com/selefra/selefra

Selefra-How to Read Code to Provide for Rule Use

Felix — Thu, 29 Jun 2023 02:30:58 +0000

Selefra Code Reading

In the previous article, we discussed how Selefra utilizes Provider to fetch data. Now, let's delve into the details of how Selefra reads YAML code.

Code Block Reading Methods

As a Policy-as-Code tool, Selefra plays a crucial role in code reading. Simple YAML parsing is not sufficient due to the interdependencies and associations between code blocks. We need to read different sections of code separately, establish connections based on keywords, and merge them accordingly. Additionally, it's essential to perform validity checks on the code and report any errors in location and content. Therefore, we need to handle this functionality with special treatment.

Let's first understand the basic methods for reading code blocks and performing validity checks during the reading process.

Selefra and Provider Modules

Selefra and Provider modules are relatively straightforward, with no complex compliance relationships. We only need to define the structure and perform checks. Here's an example code snippet showcasing the Provider module:

// Insert code model here
type SelefraBlock struct {

    // Name of project
    Name string `yaml:"name,omitempty" mapstructure:"name,omitempty"`

    // selefra CloudBlock-related configuration
    CloudBlock *CloudBlock `yaml:"cloud,omitempty" mapstructure:"cloud,omitempty"`

    OpenaiApiKey string `yaml:"openai_api_key,omitempty" mapstructure:"openai_api_key,omitempty"`
    OpenaiMode   string `yaml:"openai_mode,omitempty" mapstructure:"openai_mode,omitempty"`
    OpenaiLimit  uint64 `yaml:"openai_limit,omitempty" mapstructure:"openai_limit,omitempty"`

    // The version of the cli used by the project
    CliVersion string `yaml:"cli_version,omitempty" mapstructure:"cli_version,omitempty"`

    // Global log level. This level is used when the provider does not specify a log level
    LogLevel string `yaml:"log_level,omitempty" mapstructure:"log_level,omitempty"`

    //What are the providers required for operation
    RequireProvidersBlock RequireProvidersBlock `yaml:"providers,omitempty" mapstructure:"providers,omitempty"`

    // The configuration required to connect to the database
    ConnectionBlock *ConnectionBlock `yaml:"connection,omitempty" mapstructure:"connection,omitempty"`

    *LocatableImpl `yaml:"-"`
}

We have defined the structure, and for each code block, we have defined the parseSelefraBlock method within the YamlFileToModuleParser structure to load and validate the Selefra code blocks.

func (x *YamlFileToModuleParser) parseSelefraBlock(selefraBlockKeyNode, selefraBlockValueNode *yaml.Node, diagnostics *schema.Diagnostics) *module.SelefraBlock {

    blockPath := SelefraBlockFieldName

    // Type check
    if selefraBlockValueNode.Kind != yaml.MappingNode {
        diagnostics.AddDiagnostics(x.buildNodeErrorMsgForMappingType(selefraBlockValueNode, blockPath))
        return nil
    }

    toMap, d := x.toMap(selefraBlockValueNode, blockPath)
    diagnostics.AddDiagnostics(d)
    if utils.HasError(d) {
        return nil
    }

    selefraBlock := module.NewSelefraBlock()
    for key, entry := range toMap {
        switch key {

        case SelefraBlockNameFieldName:
            selefraBlock.Name = x.parseStringValueWithDiagnosticsAndSetLocation(selefraBlock, SelefraBlockNameFieldName, entry, blockPath, diagnostics)

        // Omit some code
        default:
            diagnostics.AddDiagnostics(x.buildNodeErrorMsgForUnSupport(entry.key, entry.value, fmt.Sprintf("%s.%s", blockPath, key)))
        }
    }

    if selefraBlock.IsEmpty() {
        return nil

    }

    // Set code location
    x.setLocationKVWithDiagnostics(selefraBlock, "", blockPath, newNodeEntry(selefraBlockKeyNode, selefraBlockValueNode), diagnostics)

    return selefraBlock
}

During the parsing of each data, we use predefined validation methods. Taking the parseStringValueWithDiagnosticsAndSetLocation method as an example:

func (x *YamlFileToModuleParser) parseStringValueWithDiagnosticsAndSetLocation(block module.Block, fieldName string, entry *nodeEntry, blockBasePath string, diagnostics *schema.Diagnostics) string {
    valueString := x.parseStringWithDiagnostics(entry.value, blockBasePath+"."+fieldName, diagnostics)

    if entry.key != nil {
        x.setLocationWithDiagnostics(block, fieldName+module.NodeLocationSelfKey, blockBasePath, entry.key, diagnostics)
    }

    x.setLocationWithDiagnostics(block, fieldName+module.NodeLocationSelfValue, blockBasePath, entry.value, diagnostics)

    return valueString
}

func (x *YamlFileToModuleParser) setLocationWithDiagnostics(block module.Block, relativeYamlSelectorPath, fullYamlSelectorPath string, node *yaml.Node, diagnostics *schema.Diagnostics) {
    location := module.BuildLocationFromYamlNode(x.yamlFilePath, fullYamlSelectorPath, node)
    err := block.SetNodeLocation(relativeYamlSelectorPath, location)
    if err != nil {
        diagnostics.AddErrorMsg("YamlFileToModuleParser error, build location for file %s %s error: %s", x.yamlFilePath, fullYamlSelectorPath, err.Error())
    }
}

In this method, we validate the code value's validity and report any errors and modification suggestions to the user when it's invalid.

Complex Code Block: Module Block

In the Module block, we require additional method invocations for loading. We load sub-modules based on the uses field within the code block and establish associations using names and parent-child module pointers for subsequent code execution.

// Load sub-modules
subModuleSlice, loadSuccess := x.loadSubModules(ctx, finalModule.ModulesBlock)
if !loadSuccess {
    return nil, false
}
finalModule.SubModules = subModuleSlice
finalModule.Source = x.options.Source
finalModule.ModuleLocalDirectory = x.options.ModuleDirectory
finalModule.DependenciesPath = x.options.DependenciesTree

These are the key operations involved in reading Selefra code. In the next article, we will explore how Selefra executes rules, queries non-compliant entries, and delivers the expected results to the user.

Thank you for reading!

Please follow our project and provide your star: https://github.com/selefra/selefra

How to spot and troubleshoot AWS S3 bucket object traversal issues

Felix — Sun, 25 Jun 2023 14:17:26 +0000

Introduction

The object traversal issue in AWS S3 buckets is a common problem that can be caused by two main reasons: incorrect ACL (Access Control List) configurations and incorrect policy configurations. In the following sections, we will provide a detailed explanation of how to detect and prevent these misconfigurations.

Introduction to Object Traversal Issue
Let's consider a storage bucket where, upon opening it, we can see the following content:

As we can observe, the bucket directly lists the files it contains, indicating the presence of an object traversal issue. Now, let's delve into why this issue occurs.

Incorrect ACL Configuration When the ACL of a storage bucket is configured to provide "List" permissions to everyone, it results in an object traversal problem.

Incorrect Policy Configuration In addition to ACL misconfigurations, incorrect policy configurations can also lead to the same problem. Here's an example of a policy with an erroneous configuration:

{
    "Version": "2012-10-17",
    "Id": "test",
    "Statement": [
        {
            "Sid": "test",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::selefra-test-xxxx"
        }
    ]
}

In this policy, all users are granted "s3:ListBucket" permissions for the "selefra-test-xxxx" bucket, which also leads to object traversal issues.

Fixing the Object Traversal Issue
To fix the issue caused by ACL misconfiguration, it is sufficient to uncheck the "List" option for "Everyone." For incorrect policy configurations, it is recommended to follow the principle of least privilege, granting specific permissions to designated users instead of providing access to all users.

Quickly Discovering Object Traversal Issues Using Selefra
Manually identifying these problems can be time-consuming and difficult to perform in bulk. Selefra can assist in quickly detecting these risks.

Selefra Project Repository: github.com/selefra/selefra

Regular Usage of Selefra
Let's start by installing Selefra:

brew tap selefra/tap
brew install selefra/tap/selefra
Next, create a new project folder:

mkdir selefra-test
cd selefra-test
Copy the following YAML file into this folder:

selefra:
    name: selefra-test
    connection:
      type: postgres
      username: your_username
      password: your_password
      host: 127.0.0.1
      port: 5432
      database: postgres
      sslmode: disable
    log_level: info
    providers:
        - name: aws
          source: aws
          version: v0.1.0
providers:
    - name: aws
      provider: aws
      cache: 7d
rules:
  - name: bucket_object_traversal_by_policy
    metadata:
      title: S3 bucket object traversal by policy
    query: |-
      SELECT
        DISTINCT(a1.*)
      FROM
        aws_s

3_buckets a1,
        json_array_elements(a1.policy :: json -> 'Statement') a2
      WHERE
      (
        a2 ->> 'Action' = 's3:ListBucket'
        OR a2 ->> 'Action' = 's3:List*'
        OR a2 ->> 'Action' = 's3:*'
      )
      AND a2 ->> 'Effect' = 'Allow'
      AND (
        a2 ->> 'Principal' = '*'
        OR a2 -> 'Principal' ->> 'AWS' = '*'
      )
      AND right(substring(a2 ->> 'Resource', -2), 2) <> '/*' ;
    output: "S3 bucket object traversal by policy, arn: { {.arn} }"
  - name: bucket_object_traversal_by_acl
    metadata:
      title: S3 bucket object traversal by acl
    query: |-
      SELECT
        DISTINCT(a1.*)
      FROM
        aws_s3_buckets a1,
        aws_s3_bucket_grants a2
      WHERE
       a1.selefra_id = a2.aws_s3_buckets_selefra_id
        AND a2.grantee :: jsonb ->> 'URI' = '<http://acs.amazonaws.com/groups/global/AllUsers>'
        AND a2.permission IN ('READ', 'FULL_CONTROL');
    output: "S3 bucket object traversal by acl, arn: { {.arn} }"
This configuration consists of three modules: "selefra," "providers," and "rules." Under the "selefra" module, configure your PostgreSQL database connection details in the "connection" block. The "providers" module includes the AWS provider and its cache duration. The "rules" module contains the configurations related to the detection rules. The "title" field represents the title of the detection strategy, and the "query" field contains the SQL query used to execute the detection strategy by querying the database for at-risk resources.

Before starting the detection, configure your AWS credentials using the following command:

aws configure

Then, run the following command to execute Selefra:

selefra apply
Selefra will initiate the detection process, and you will receive results similar to the example below:

In the results, you can identify the at-risk storage buckets. In addition to the above method, Selefra also integrates the chatGPT feature, allowing you to discover risk points by directly querying Selefra.

Selefra's GPT Feature

Similar to the previous steps, create a new folder and copy the following YAML file into it:

selefra:
    name: selefra-test
    connection:
      type: postgres
      username: yourusername
      password: yourpassword
      host: 127.0.0.1
      port: 5432
      database: postgres
      sslmode: disable
    log_level: info
        openai_api_key: your_openai_api_key
    openai_mode: gpt-4
    openai_limit: 10
    providers:
        - name: aws
          source: aws
          version: v0.1.0
providers:
    - name: aws
      provider: aws
      cache: 7d
rules:

In this configuration, you need to provide your own OpenAI API key and specify whether to use GPT-4 or GPT-

3.5. The "rules" block can be left empty as it will be automatically generated by AI.

Before starting the detection, configure your AWS credentials, and then you can use the GPT feature as follows:

selefra gpt "Query S3 Buckets that allow list objects"
The output will provide you with the discovered risks in your cloud environment using just a simple query.

As shown above, you can easily identify risks in your cloud environment with just a single query, making it very convenient.

Conclusion

Object traversal issues in S3 storage buckets are common and important to address. Through this article, I hope to help you understand and mitigate the object traversal issues in AWS S3 storage. With the help of Selefra, we can make the cloud more secure.

Selefra-How Provider pulls data to Selefra

Felix — Sun, 25 Jun 2023 04:55:35 +0000

Introduce

In Selefra, we use providers to collect data for Policy as Code. Providers are code snippets written based on the official API documentation provided by cloud services or Software as a Service (SaaS) providers. When using Selefra, users write provider code that is executed to send requests to the cloud and retrieve data. The returned data is then stored in PostgreSQL databases in the predefined format.

AWS Provider Get Resource

Taking AWS resources as an example, users can choose specific AWS services for data collection. Here's an example of using the AWS SDK ("github.com/aws/aws-sdk-go-v2/aws/service/s3") for data collection:

func (x *TableAwsS3BucketsGenerator) GetDataSource() *schema.DataSource {
    return &schema.DataSource{
        Pull: func(ctx context.Context, clientMeta *schema.ClientMeta, client any, task *schema.DataSourcePullTask, resultChannel chan<- any) *schema.Diagnostics {

            diagnostics := schema.NewDiagnostics()

            cl := client.(*aws_client.Client)
            svc := cl.AwsServices().S3
            response, err := svc.ListBuckets(ctx, nil, func(options *s3.Options) {
                options.Region = listBucketRegion(cl)
            })
            if err != nil {
                return schema.NewDiagnosticsErrorPullTable(task.Table, err)
            }

            var wg sync.WaitGroup
            buckets := make(chan types.Bucket)
            errs := make(chan error)
            for i := 0; i < fetchS3BucketsPoolSize; i++ {
                wg.Add(1)
                go fetchS3BucketsWorker(ctx, client, buckets, errs, resultChannel, &wg)
            }
            go func() {
                defer close(buckets)
                for _, bucket := range response.Buckets {
                    select {
                    case <-ctx.Done():
                        return
                    case buckets <- bucket:
                    }
                }
            }()
            done := make(chan struct{})

            go func() {
                for err = range errs {
                    diagnostics.AddErrorPullTable(task.Table, err)
                }
                close(done)
            }()
            wg.Wait()
            close(errs)
            <-done

            return diagnostics
        },
    }
}

In this example, we use the AWS S3 SDK to retrieve the list of AWS S3 buckets. We leverage the ListBuckets method and use goroutines to concurrently fetch detailed information for each bucket. The obtained information is then stored in our predefined data structure. The function returns a pointer to the schema.DataSource type, which contains the method for retrieving the bucket list.

AWS Provider creates a local data storage structure
Of course, you need to have some understanding of the API for the data you want to collect. You need to define the table structure where you want to store your data in advance.

func GenTables() []*schema.Table {
    return []*schema.Table{
        table_schema_generator.GenTableSchema(&amp.TableAwsAmpWorkspacesGenerator{}),
        table_schema_generator.GenTableSchema(&eks.TableAwsEksClustersGenerator{}),
        // Omitted 430 lines of code
        table_schema_generator.GenTableSchema(&kinesis.TableAwsKinesisStreamsGenerator{}),
        table_schema_generator.GenTableSchema(&cloudfront.TableAwsCloudfrontCachePoliciesGenerator{})
    }
}

The table structure could be quite large, but we can use GPT to handle it. On top of that, we only need to do some manual auditing.

You can write provider code according to your needs to retrieve data for other AWS resources, such as S3 buckets, RDS databases, Lambda functions, etc. Refer to the AWS official API documentation to understand the API methods and parameters for each service and write the provider code accordingly.

By using provider code, Selefra sends requests to AWS services and

retrieves the required data. This data is then stored in the PostgreSQL database in the predefined format for subsequent policy analysis and decision support.

Please note that the provided example code is for demonstration purposes only, and the actual provider code may vary depending on the AWS service and API used. We recommend referring to the AWS official documentation for writing provider code specific to the desired service.

Conclusion

This is a detailed explanation of how data is retrieved from cloud resources using providers in Policy as Code products like Selefra. There are many details involved, such as retry mechanisms and error handling, which are specific to the business logic and can be addressed accordingly. You can also visit our official repository to view the open-source provider code. If you would like to stay updated with our team and experience our open-source product Selefra:

GitHub: https://github.com/selefra/selefra

Slack: https://selefra.io/community/join

Selefra Release Note v0.2.2

Felix — Tue, 20 Jun 2023 02:47:46 +0000

Date: June 11, 2023

We are excited to announce the release of Selefra v0.2.2, an open-source Policy-as-Code (PaC) product. This version introduces several bug fixes and enhancements to improve your experience with Selefra.

Here are some key updates:

Added the "--dit" option to define the result output directory.
Addressed bug #19, where Selefra GPT returned a "mode not found" error.
Remediation in rules now supports plain text input and is no longer enforced to be a file path.
Adjusted output colors for improved readability.
Where Selefra GPT functionality informed users of a syntax error.

We appreciate your feedback and contributions to the Selefra community. Your input in identifying and resolving these issues is invaluable. If you encounter any further bugs or have suggestions for future updates, please do not hesitate to reach out to us.

To get started with Selefra v0.2.2, please visit our official repository (Github: https://github.com/selefra/selefra) and follow the installation instructions. Remember to update your current version to benefit from the latest enhancements.

Thanks for your continued support, and we hope you find Selefra v0.2.2 to be a valuable PaC implementing tool in your projects.

Enjoy!

The Selefra Team

Github:https://github.com/selefra/selefra

How does policy as code work

Felix — Thu, 15 Jun 2023 04:48:05 +0000

What is Policy as Code?

Policy as Code is a method of writing security and compliance policies as executable code. These policies can be used alongside an organization's applications and infrastructure to ensure they meet security and compliance requirements.

Compared to traditional security and compliance approaches, Policy as Code offers a more efficient and precise way of enforcement. It also enables organizations to automate security and compliance policies within their development and deployment processes, improving efficiency and reducing operational risks.

How does Policy as Code work?

Policy as Code relies on three key components:

Data

To analyze existing cloud, SaaS, or infrastructure environments, it's necessary to obtain the underlying data. In the context of Policy as Code tools, data is typically fetched using providers. In the case of Selefra, providers are used to retrieve data from various cloud service providers. During the initial stage of running Selefra, open-source (or user-defined) providers are used to fetch and store the cloud resources' data in a designated PostgreSQL database. This data serves as the basis for policy writing and analysis in subsequent steps.

Policies

Policies are written as code that simulates decision-making behavior. These rules describe security, compliance, or best practice requirements. Policies can be based on industry standards (such as CIS Benchmarks) or internal organizational policies. They can be expressed using natural language, domain-specific languages (DSLs), or code. In Selefra, policies are defined in a YAML format with placeholders and SQL statements.


rules:
  - name: bucket_acl_publicly_writeable
    query: |-
      SELECT
        DISTINCT(a1.*)
      FROM
        aws_s3_buckets a1,
        aws_s3_bucket_grants a2
      WHERE
       a1.selefra_id = a2.aws_s3_buckets_selefra_id
        AND a2.grantee :: jsonb ->> 'URI' = '<http://acs.amazonaws.com/groups/global/AllUsers>'
        AND a2.permission IN ('WRITE_ACP', 'FULL_CONTROL');
    output: "S3 bucket ACL publicly writable, ARN: { {.arn} }"

In the above example, Selefra retrieves the AWS data based on the user's configuration and stores it in a specified format. Then, it applies the defined rules to identify non-compliant items.

Execution

Once the data and policies are in place, a tool is needed to execute the policies and query for potential security risks or compliance violations. In the case of Selefra, the policies are interpreted as executable code. The query section of the policy is executed, and the results are formatted and outputted using the output section. Users can view the identified issues with their cloud resources in the console.

Conclusion

That concludes a brief explanation of how a Policy-as-Code product works. Each step of implementation requires significant code design. In our upcoming articles, our team will provide detailed explanations of the architecture design and code implementation for each step. If you're interested in following our team and experiencing our open-source product, Selefra, please visit:

GitHub: https://github.com/selefra/selefra

Slack: https://selefra.io/community/join

Thank you very much for reading!

Policy-as-Code: Enhancing Security and Compliance in a New Trend

Felix — Mon, 12 Jun 2023 06:45:05 +0000

Introduction:

In the ever-evolving landscape of cybersecurity and regulatory compliance, organizations are constantly seeking innovative approaches to bolster their security posture and ensure adherence to industry regulations. One such emerging trend is the implementation of Policy-as-Code, a powerful methodology that combines policy enforcement with the flexibility and scalability of code. By embracing Policy-as-Code, organizations can enhance their security and compliance measures while adapting to the dynamic nature of the modern digital environment.

Here are two scenarios in below:

Scenario 1 - PolicyAsCode：

rules:
  - name: bucket_acl_publicly_readable
    query: |-
      SELECT
        DISTINCT(a1.*)
      FROM
        aws_s3_buckets a1,
        aws_s3_bucket_grants a2
      WHERE
        a1.selefra_id = a2.aws_s3_buckets_selefra_id
        AND a2.grantee :: jsonb ->> 'URI' = '<http://acs.amazonaws.com/groups/global/AllUsers>'
        AND a2.permission IN ('READ_ACP', 'FULL_CONTROL');
    output: "S3 bucket acl public readable, arn: { {.arn} }"

With the assistance of the PolicyAsCode tool and its provided basic configuration documentation, we can define the query criteria we need. The PolicyAsCode tool will automatically retrieve the resources and conduct an analysis, enabling us to quickly identify any problematic cloud assets and receive recommendations for remediation.

Scenario 2 - Traditional Approach In the traditional approach, addressing the same issue would involve manually inspecting the S3 configurations across multiple cloud assets in AWS. This process would require logging into different AWS accounts and performing repetitive manual tasks, making it cumbersome and time-consuming.

Enhancing Security:

Policy-as-Code serves as a robust framework for enforcing security policies throughout an organization's software systems. By translating policies into executable code, it enables automated enforcement, reducing the risk of human error and ensuring consistent application of security measures. With Policy-as-Code, organizations can define and enforce access controls, authentication mechanisms, encryption standards, and other security policies across their entire infrastructure, providing a comprehensive and proactive security approach.

Streamlining Compliance:

Compliance with industry regulations and standards is a critical aspect of business operations. Policy-as-Code simplifies the compliance process by codifying regulatory requirements into executable policies. This ensures that compliance checks are automated and consistently applied, reducing the burden of manual audits and minimizing the risk of non-compliance. With Policy-as-Code, organizations can easily track and demonstrate compliance with regulations such as GDPR, HIPAA, PCI-DSS, and others. This facilitates smoother audits, mitigates penalties, and ensures a robust compliance framework.

Agility and Scalability:

In today's fast-paced digital landscape, organizations need agile and scalable solutions to keep up with evolving threats and changing compliance requirements. Policy-as-Code offers the flexibility to adapt policies quickly and efficiently. Code-based policies can be version controlled, tested, and easily modified, allowing organizations to respond rapidly to emerging security risks and compliance updates. This agility and scalability enable organizations to maintain robust security and compliance measures in dynamic environments.

Collaboration and Transparency:

Policy-as-Code fosters collaboration among teams responsible for security and compliance. By representing policies as code, it promotes transparency and enables collaborative development and review processes. Different stakeholders can contribute to policy development, ensuring a collective understanding and ownership of security and compliance requirements. This collaborative approach facilitates knowledge sharing, promotes best practices, and empowers teams to continuously improve security and compliance measures.

Conclusion:

As security threats and regulatory complexities continue to evolve, organizations must adopt innovative approaches to strengthen their security and compliance posture. Policy-as-Code emerges as a powerful tool, enabling automated policy enforcement, streamlining compliance processes, and providing agility in an ever-changing landscape. By embracing Policy-as-Code, organizations can enhance their security measures, ensure regulatory compliance, and stay ahead in the face of emerging security challenges. The future of security and compliance lies in the fusion of policy enforcement and code, making Policy-as-Code an essential component of any modern organization's security strategy.

Thanks for your reading!

Selefra: https://github.com/selefra/selefra

SelefraTeam - Revolutionizing Security and Resource Management with PolicyAsCode

Felix — Fri, 02 Jun 2023 03:41:26 +0000

SelefraTeam is a dynamic team of dedicated professionals driven by a shared passion for innovation and excellence. We specialize in developing cutting-edge solutions to address the ever-evolving challenges of security and resource management in the complex landscape of cloud services and Software-as-a-Service (SaaS).

At the heart of our offerings lies PolicyAsCode, a groundbreaking tool that seamlessly integrates security and flexibility. Powered by the efficient and robust Go programming language, our open-source project on GitHub (https://github.com/selefra/selefra) fosters a vibrant and inclusive community, encouraging active participation and collaboration.

Transparency, trustworthiness, and collaboration are the cornerstones of our philosophy. We take pride in hosting over 30 providers, offering users a diverse range of options. We also foster an environment where users are empowered to develop and contribute their own providers, enriching the open-source ecosystem for the benefit of all.

What sets our PolicyAsCode tool apart is its ability to express rules in a concise and clear YAML format, enabling their seamless application across different cloud services. Whether it's performing security checks, optimizing cost configurations, or conducting architectural reviews, our tool provides robust and flexible functionalities to meet the ever-changing demands of users.

However, our vision extends beyond delivering a simple solution. We strive to empower each user, enabling them to independently craft policies that align with their unique cloud service requirements. In addition to utilizing our comprehensive library of open-source policies for detection and review, users have the freedom to customize policies to suit their specific needs, ensuring personalized security management and resource optimization.

As SelefraTeam, we are committed to fostering collaboration and actively listening to our users. By understanding their needs and continuously iterating and improving our tool's performance and features, we aim to deliver exceptional products and services. Through strong partnerships with our users and the wider open-source community, we believe in collective success and aim to redefine the landscape of cloud services and SaaS security.

When you choose SelefraTeam, you choose industry-leading expertise, unwavering professionalism, and a commitment to delivering secure, efficient, and flexible solutions. Whether you are a large-scale organization or an individual developer, we are here to be your trusted partner in navigating the realms of cloud services and SaaS security in the digital era. Join us on this exciting journey as we shape the future of security and resource management with PolicyAsCode.

GitHub:https://github.com/selefra/selefra

DEV Community: Felix

Introduction to EC2 instance metadata and risk prevention

Introduction

Risks

Risk Detection

Manual Assessment

Using Selefra for Assessment

Using Selefra GPT for Assessment

Conclusion

Why Choose PGSQL as the Database for Resource Storage

Analyzing the Ideal Database Choice for Selefra's Policy-as-Code Tool

Introduction

Scalability

Data Consistency

Powerful Query Capabilities

Security

Mature Ecosystem

Conclusion

recommended reading

A Better Version Is Released - Selefra v0.2.3

Release Note - Selefra v0.2.3

Date: June 30, 2023

Summarize

recommended reading

How to spot and fix issues with publicly accessible AWS S3 buckets

Introduction

Understanding the Problem

Remediation

Using Selefra for Quick Discovery

Typical Usage of Selefra

Selefra GPT Feature

Conclusion

How Selefra Combines GPT for Multi-Cloud Security Scanning

Introduction

Expert Simulation

Resource Acquisition

Resource Analysis

Summary

Selefra-How to Read Code to Provide for Rule Use

Selefra Code Reading

Code Block Reading Methods

Selefra and Provider Modules

Complex Code Block: Module Block

How to spot and troubleshoot AWS S3 bucket object traversal issues

Introduction

Selefra's GPT Feature

Selefra-How Provider pulls data to Selefra

Introduce

AWS Provider Get Resource

Conclusion

Selefra Release Note v0.2.2

How does policy as code work

What is Policy as Code?

How does Policy as Code work?

Data

Policies

Execution

Conclusion

Policy-as-Code: Enhancing Security and Compliance in a New Trend

Introduction:

Here are two scenarios in below:

Enhancing Security:

Streamlining Compliance:

Agility and Scalability:

Collaboration and Transparency:

Conclusion:

SelefraTeam - Revolutionizing Security and Resource Management with PolicyAsCode