DEV Community: Takashi Narikawa

My Python Project Development Guide ver.202412

Takashi Narikawa — Sun, 14 May 2023 11:17:52 +0000

What is this

The guide to set up a Python project for production deployment.
This guide assumes you are using uv for package management and adheres to the rules specified.
By following this guide, you can set up a Python project for production deployment using uv as the package manager, enforce code quality with linters and formatters, automate checks with GitHub Actions, configure your IDE to maintain consistency, and write comprehensive unit tests to ensure the project's reliability throughout the development process.
We recommend using Python version 3.10 or higher for your project.
The majority of this document was created using ChatGPT-4.
Sample Project
- https://github.com/fukubaka0825/my-python-guide

Steps

1. Project Setup/ Set up Local Environment with uv

Create a project directory and navigate to it:

mkdir ${project_path}
cd ${project_path}

uv allows you to manage local Python versions without the need for pyenv. It uses a .python-version file for project-specific Python version management.

uv python install ${python_version}

Replace ${python_version} with the desired version, e.g., 3.10.2. This command will create a .python-version file in the project directory.
Ensure that all team members use the same Python version specified in the .python-version file.
Copy the provided file to the project directory and install dependencies:
- Copy pyproject.yml example and arrange by yourself
- For the local environment, use uv to create a .venv virtual environment in the project root.
- Execute the following commands to set up the virtual environment and install the required packages

uv venv
uv sync
# Run command in the created environment.
uv run <YOUR_COMMAND>

In the pyproject.toml file, we configure various modules for linting, formatting, and task running. Here's an explanation of these modules:

Taskipy

Taskipy is a task runner that allows you to automate various tasks in your Python project. In this project, we use Taskipy to define and run tasks such as formatting, linting, and testing. The tasks are configured in the pyproject.toml file under [tool.taskipy.tasks].

For more information about Taskipy, check out the official documentation:

Linters and Formatters

This project uses several linters and formatters to maintain code quality and consistency:

Black: A code formatter that automatically formats your code according to the PEP 8 style guide. The configuration is set in the pyproject.toml file under [tool.black].

Ruff: A linter that checks your code for style guide violations, potential bugs, and other issues. This covers so many rules inluding flake8’s one and isort’s one. The configuration is set in the pyproject.toml file under [tool.ruff].

Mypy: A static type checker that helps catch potential bugs and ensures that your code is type-safe. The configuration is set in the pyproject.toml file under [tool.mypy].

These linters and formatters are integrated into the project's tasks and the IDE settings, ensuring a consistent code style across the entire project.

2. Create Directory Structure

Create the necessary directories and files:

mkdir assets　# you can write main codes in this directory
mkdir tests # you can write test codes in this directory
touch README.md

3. Write Sufficient Unit Tests with pytest

Ensure that you write comprehensive unit tests using pytest for all critical components of your project. Adequate test coverage helps maintain code quality and reduces the likelihood of introducing bugs during development or refactoring.

To write a unit test, create a unit test, create a new file in the tests directory with a name that follows the pattern test_*.py.

Within the test file, import the necessary modules and define test functions that follow the pattern def test_*():.

Use pytest's built-in assert statement to compare expected and actual values.

To run tests, execute the following command:

uv run task test

Ensure that all tests pass before committing changes and creating a pull request.

4. Configure IDE Settings

Reflect the contents of the provided pyproject.toml file in the settings file related to your IDE.

For example, if you are using VSCode, add the following settings to .vscode/settings.json:

{
  "python.formatting.provider": "black",
  "python.formatting.blackArgs": ["--line-length", "102"],
  "python.linting.flake8Enabled": true,
  "python.linting.mypyEnabled": true,
  "python.sortImports.args": ["--profile=black", "--line-length=102"],
  "editor.codeActionsOnSave": {
    "source.organizeImports": true,
    "source.fixAll": true
  }
}

With these settings, VSCode will automatically format and lint code using Black, Flake8, Mypy, and isort.
If you use IntelliJ/Pycharm, Please use File Watcher function for auto-running uv run task fmt_lint

5. Set up CI(Auto lint/test) with GitHub Actions

Create the .github/workflows directory in the project root:

mkdir -p .github/workflows

Create a test_and_lint.yml file and add the following content:

name: python lint and test

on:
  pull_request:
    branches:
      - main
    paths:
      - "YOUR_PROJECT/**.py"
      - "tests/**.py"
      - ".github/workflows/test_and_lint.yml"

jobs:
  lint_and_test:
    name: Lint and Test
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.11","3.10","3.9"]

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install uv
        uses: astral-sh/setup-uv@v4
        with:
          enable-cache: true

      - name: Install Dependencies
        run: |
          uv sync

      - name: Python Test
        run: |
          uv run task test

      - name: Python Lint
        continue-on-error: false
        run: uv run task lint

sample
- https://github.com/fukubaka0825/my-python-guide/blob/main/.github/workflows/docker-build-push-xxxxxxxxxx.yml
This configuration will automatically run uv run task fmt_lint and uv run task test on pushes and pull requests to the main branch.

6. Set up Stage/Prod Environment with Dockerfile

For the stage/prod environment, use a Dockerfile to build the execution environment. First, export the project's dependencies to a requirements.txt file using the following command:

uv export --no-dev --no-hashes --format requirements-txt > requirements.txt

In the Dockerfile, use pip install to install the dependencies from the requirements.txt file. Ensure that the specified Python version in the Dockerfile is the same as the one used in the local environment.

sample Dockerfile
- https://github.com/fukubaka0825/my-python-guide/blob/main/Dockerfile

By following these rules, you can ensure a consistent development environment for all team members and maintain a clean separation between local and stage/prod environments.

Additionally, it is recommended to integrate the configuration of this process into the GitHub Actions YAML for the next 7 Continuous Deployment (CD) steps, as it eliminates the need to work locally.

7. Set up CD(Atuo build Dockerfile) with Github Actions

sample
- https://github.com/fukubaka0825/my-python-guide/blob/main/.github/workflows/test-and-lint-xxxxxxx.yml

Appendix

Why are we organizing a tech conference called SRE NEXT 2022?

Takashi Narikawa — Sun, 27 Feb 2022 13:23:36 +0000

Japanese Edition:
https://blog.sre-next.dev/entry/2022/02/06/190047

Introduction

Hello, I'm nari/wapper, the organizer of SRE NEXT 2022.

The other day, the SRE NEXT official Twitter account announced that SRE NEXT 2022 will be held online on May 14 and 15, and the official website has also been published!

In this post, I will explain why we are holding SRE NEXT 2022 online and what kind of conference we want to hold.

What is SRE NEXT?

SRE NEXT is a conference for engineers who are deeply interested in reliability practices, and is run and organized by the members of SRE Lounge, another community-based SRE study group. This community is organized with mainly Japanese.

The first SRE NEXT 2020 was held in January 2020 in Tokyo, and the second is scheduled to be held this year in 2022. It is a conference for anyone interested in getting reliability with users, not just those working with SRE role. SRE is an idea and a practice, and there are many companies that have achieved this without having it as a role.

The tickets of the first conference were sold out and more than 50 recap articles were wrote and published after the event.

Why we hold SRE NEXT 2022

With the last SRE NEXT 2020, we believe that I was able to gather cases of SRE practices in the web domain of certain size companies and spread them throughout Japan.
In the years since then, the number of topics and events related to SRE has been increasing, and the focus of attention has been growing up day by day. In particular, I've heard a lot about SLI/SLO design and operation for general availability in web applications.
In order to accelerate this trend and to play a role in spreading SRE practices to a wider range of people and domains, we decided to hold SRE NEXT 2022 with the theme "SRE DIVERSITY".

For example, in terms of variety of scales/phases, I had the impression that many of the presenters and participants of SRE NEXT 2020 were engaged in in-house products of a certain big scale. I think it will be more interesting if we can hear about SRE practices at more different scales and forms. Also, the fact that the best way of SRE practices is different depending on the phase of the product has been discussed in various places, and is even mentioned and cited in "Team Topology".
Also, in terms of variety of domains/areas, We want to adopt proposals and sessions in more diverse areas like "Jim Thomson and David Laing of Pivotal at SREcon19, Extending the Error Budget Model to Security and Feature Freshness" and the story of SRE practices and social movements in Seeking SRE and
sessions related to AIOps, which is listed as a major monitoring component in catchpoint's SRE 2021 Report, and so on. I think it would be good to have more discussions on wide range of areas.

With these in mind, we decided to hold SRE NEXT 2022 in order to provide an opportunity for the spread of more diverse SRE practices by gathering SRE practices in a wide range of industries, domains, and phases, from startups to large corporations, especially in Japan.
SRE book co-author Naill Murphy's keynote at SRECon21, "Don't Follow Leaders or "All Models Are Wrong (and So Am I)", he said that the absolute influence of SRE books has led to a cessation of thinking and a lack of innovation, and that what is important is to evolve these models with the times.
I believe that the trial and error of gaining reliability with a wider variety of users will provide participants with a lot of insights and learning, and give them a chance to think over their SRE practices and think about the "NEXT of SRE ".

What kind of conference we want to hold

The last event was well attended by mainly SREs.
But, As for 2022 event, I would like to invite people not only from SREs but also from those who are struggling to achieve reliability in their daily lives.
Also, as with the SRE Lounge, we would like to provide a place for interactive discussions, so we are working on providing a reception and an online venue.
In addition, since this event will be held online, it will be a unique opportunity for those who were unable to attend last time to participate and interact with us.

Our goal is to create an event that will help to democratize and generalize the idea and practice of SRE, so that it is not just for people with limited responsibilities and areas, but can be talked about in a variety of areas.

Conclusion

In addition to providing a lineup of attractive sessions, we are also considering and working on various events, receptions, online venues, and other contents on the same day. ), so please follow us on Twitter and stay tuned for more information.
As in the previous event, we are looking forward to working with the participants and sponsors to make the event even better, and we look forward to your continued support of SRE NEXT.

Here is Session Proposal Form for English speaking folks

Introducing Conftest and setting up CI with Github Actions to automate reviewing of Terraform code

Takashi Narikawa — Tue, 28 Dec 2021 12:48:27 +0000

日本語版はこちら

Introduction

Hi, I’m nari/wapper from SRE team at eureka, Match Group.👋

In our SRE Team, we are planning to set micro-services ready as one of the major keywords for next year ro prevent ourselves from becoming a bottleneck. (We are not going to rebuild our system as micro-services, though).

With that, as well as the Production Readiness check, I've started to try to use Policy as Code with Conftest, which is an important piece to promote transferability/Self Service, and I'll write about it as a memorandum.

In this article, I'll show you how to write a policy for resource tag rules (AWS reousrce) in Terraform with Rego, write tests for the policy itself, and set up the CI for its automatic execution using Github Actions.
(There is a lot of talk about writing Conftest (Rego), but not much has been written about setting up CI, and there were some points where I got stuck, so my main motivation for writing this article is to leave a reminder so that people who are about to try it don't get stuck.)

Target readers

People who have been curious about Policy as Code but needed a reason to start.
People who want to know concrete examples of how to set up Github Actions CI of Conftest.

What is OPA/Rego/Conftest???

OPA (Open Policy Agent) is an OSS general-purpose policy engine, which is currently a Graduation project of CNCF. OPA writes policies in a policy description language called Rego. OPA evaluates the policy according to the policy written in Rego and structured data such as JSON (base document), and returns the result in the same way as structured data such as JSON (virtual document).
- It just graduated from CNCF this year (2021): [https://www.infoq.com/news/2021/02/opa-cncf-graduation/:title]
- Source code: [https://github.com/open-policy-agent/opa:title]
Conftest is a tool that supports various types of file-formats such as YAML, JSON, HCL, Dockerfile, etc. based on the policy written in OPA's Rego, and can be tested easily from the command line.
- Conftest was originally a third party tool using OPA's Rego, and now it seems to be a widely used CLI tool officially under the umbrella of the OPA community!
- Source code: [https://github.com/open-policy-agent/conftest:title]
- This time, we will adopt this tool because it is supposed to be used in CI.

Write policy rules in Rego and set up CI with Github Actions

All sample code can be found here: https://github.com/fukubaka0825/terraform-conftest-with-acitions-ci-sample

In this article, I'd like to write a policy rule to make sure that a resource created by terraform has a minimum tag (name/owner/description) and that the importance of the data (high/middle/low) is set to the tag named data when it is a data store resource. We will use the result of the terraform plan converted to json as the input value so that we can test it with the policy rule in Rego we are about to write, and also test the policy rule itself. (The sample we are going to describe assumes the use of AWS)

0.Prerequisites

0.1.Set up Conftest (OPA/Rego)

If you are a Mac OS user, you can easily install it with brew. Users of other OS can also install it by referring to the README.



brew install conftest

# also install OPA
brew install opa

As for the editor plugin, you can use opa plugin for IntelliJ or opa plugin if you are IntelliJ/VSCode user.
- I'm not sure but other editors may have them too, so try to find them and add them.

0.2.Structure of Terraform Plan results

With this structure, we will write a logic in Rego to check the content of this data as input.

1. Try to write Terrafom resource tag rules in Conftest/Rego

First, we will write the entry point in main.rego.



# policy/main.rego


package main //❶

import data.tags_validation //❷

#####################################
# Policy as Code Rules
#####################################
deny_tags_contain_minimum_set[msg] {
    # Only target resources that have been changed/added.
    changeset := resources_not_no_op_action[_] //❸

    not tags_validation.contain_proper_keys(changeset.change.after.tags)

    msg := sprintf("Invalid tags (missing minimum required tags: name,owner,description) for the following resource: %v", [changeset.address]) //❹
}

deny_data_store_data_tag_is_proper[msg] {
    # Only target resources that have been changed/added.
    changeset := resources_not_no_op_action[_]

    # Only when resource_type is a data source/store type that can contain sensitive information
    tags_validation.is_data_tag_required_target_resource(changeset.type)

    tags_validation.not_has_proper_data_tag(changeset.change.after.tags.data)

    msg = sprintf("data tag needs to be set to one of [low,high,middle] resource: %v", [changeset.address])
}

#####################################
# Utils
#####################################

resources_not_no_op_action= {resource | resource := input.resource_changes[_]; resource.change.actions[_] != "no-op"} //❺

resources_with_type(resources, type) = all {
    all := [item | item := resources[_]; item.type == type]
}

❶ policy always requires a package clause. You can use any package name to divide the namespace.

❷ You can use the import clause to import a different namespace and use the resources in that namespace. This time, you can write your own logic in the tags_validation namespace and use it in main.

❸ Here, narrow down the resources to those whose action is not "no-op", and iterate through those resources (expression: [_]) to check for resources that violate the policy. This will ensure that only resources that are newly modified/added are targeted. In addition, since this operation is basically used in all policy rules, we have made it a common function and cut it out.

❹ Multiple evaluation expressions can be written in the same rule, and since they are evaluated in AND fashion, only when all of them are true, deny/violation is judged, and error messages are registered in the error message array for each of these policy rules and output at test execution time.

❺ In Rego, the familiar Python notation for inclusion can be used. The syntax is [ | ], and only the resources that match the rule are returned.




# policy/tags.rego

package tags_validation

minimum_tags = {"name", "owner", "description"}

contain_proper_keys(tags) {
    # Subtract the key list of the given tags from the minimum tag list, and if there is no more left, you have the minimum tag.
    keys := {key | tags[key]}
    leftover := minimum_tags - keys
    leftover == set()
}

not_has_proper_data_tag(value) { // ❶
    value != "low"  
    value != "middle"
    value != "high"
}

is_data_tag_required_target_resource(type) { // ❷
    type == "aws_dynamodb_table"
}

is_data_tag_required_target_resource(type) {
    type == "aws_s3_bucket"
}

❶ The evaluation expression within the same rule will be evaluated as AND, so it will return true if it is neither "low", "middle", nor "high".

❷ If you want to write OR logic, you can do so by writing multiple processes with the same name, such as is_data_tag_required_target_resource.

2. Write tests for the rule itself written in Rego with Conftest and try to run it.

First, we will write a test for main.rego. As in Go, we will name it ${test_target_file_name}_test.rego.



# policy/main_test.rego

package main

#####################################
# Tests of Policy as Code Rules
#####################################

...

test_tags_contain_minimum_set {
    plan := `                    // ❶
      resource_changes:
        - name: case normal
          address: module.one
          type: aws_security_group_rule
          change:
            actions:
              - create
            after:
              tags:
                name: hoge
                owner: piyo
                description: for test
    `

    input := yaml.unmarshal(plan)
    deny_tags_contain_minimum_set == set() with input as input // ❷
}

...

❶ The input for the test is written in json in the official documentation, but json is hard to maintain, so it is better to manage the description in yaml and convert it to json for use.

❷ The test for the rule can be written like this. If you don't need to check the message content of the array, you can just check whether the error message array is empty or not.



# tag_rest.rego

package tags_validation

test_contain_proper_keys {
    tags := {"name": "test", "owner": "hoge", "description": "normal test"}
    contain_proper_keys(tags) // ❶
}

...

❶ The function used in the main rule is unit tested in this way, and is considered to be passed if it returns true

3.Set CI to run Conftest with Github Action

3.1.Set up CI for fmt/verify of the policy rule itself written in Conftest/Rego.



name: conftest-fmt-and-verify-all

on:
  pull_request:
    branches: [ main ]
    paths:
      - 'policy/**'
env:
  CONFTEST_VERSION: 0.28.3
jobs:
  terraform:
    name: fmt-all
    runs-on: ubuntu-latest
    defaults:
      run:
        shell: bash
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Install conftest
        run: |
          wget -O - 'https://github.com/open-policy-agent/conftest/releases/download/v${{ env.CONFTEST_VERSION }}/conftest_${{ env.CONFTEST_VERSION }}_Linux_x86_64.tar.gz' | tar zxvf -
          ./conftest --version

      - name: conftest fmt
        run: |
          git add . && ./conftest fmt ./ && git diff --exit-code ./.    // ❶

      - name: conftest verify
        run: |
          ./conftest verify ./   // ❷

❶ We don't have a flag like terraform's validation yet, so I write codes make it that the test will fail if there is a difference in the result of conftest fmt.

❷ Here, we run the test for the policy rule written in Rego that we wrote earlier.

3.2.Setting up CI to test Terraform plan results with Conftest test



name: tf-plan-apply

on:
  pull_request: 
    branches: [ main ]

env:
  TF_VERSION: 1.0.0
  CONFTEST_VERSION: 0.28.3
  WORKING_DIR: ./
jobs:
  terraform:
    name: aws-eureka-pairs-etc-s3
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Install conftest
        run: |
          wget -O - 'https://github.com/open-policy-agent/conftest/releases/download/v${{ env.CONFTEST_VERSION }}/conftest_${{ env.CONFTEST_VERSION }}_Linux_x86_64.tar.gz' | tar zxvf -
          ./conftest --version //❶

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v1
        with:
          terraform_wrapper: false //❷
          terraform_version: ${{ env.TF_VERSION }}
          cli_config_credentials_token: ${{ secrets.YOUR_CRED_NAME}}

      - name: Terraform Init ${{ env.WORKING_DIR }}
        working-directory: ${{ env.WORKING_DIR }}
        run: terraform init

      - name: Terraform Plan ${{ env.WORKING_DIR }}
        if: github.event_name == 'pull_request'
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        working-directory: ${{ env.WORKING_DIR }}
        id: plan
        run: terraform plan -out=tfplan -no-color -lock=false -parallelism=50

      - name: Convert terraform plan result to json formmat
        if: github.event_name == 'pull_request'
        id: convert
        working-directory: ${{ env.WORKING_DIR }}
        run: terraform show -json tfplan > tfplan.json

      - name: conftest test
        if: github.event_name == 'pull_request'
        id: conftest
        run: ./conftest test --no-color ${{ env.WORKING_DIR }}/tfplan.json //❸

❶ I've been using wget to install conftest since I couldn't find any properly maintained Actions.

❷ If you are using the official hashicorp/setup-terraform, be aware that this value is set to true by default and the plan results cannot be shown in the appropriate json, so you need to set it to false.

❸ Now we will test the terraform plan results using the policy rule we just wrote. By default, it refers to policy/ when reading the policies to be used. If you want to refer to a different directory, you can set it by passing -p.

Conclusion

My honest impression after playing around with Conftest/OPA/Rego is that I would like to have a schema for data definitions because I am a type enthusiast, and the logic evaluation is quite strong and expressive, so if I don't try to write it in a simple and understandable way, it will soon become impossible for the team to maintain it.

Also, It makes me comfy that I can easily write tests for Policy as Code itself. I can change, add, and refactor Rego code with confidence.

Also, I feel that each company is still searching for a strategy for directory structure of OPA/Rego, so I'd like to try and find a good place to put it.

References

Manage EKS aws-auth configmap with terraform

Takashi Narikawa — Sat, 25 Dec 2021 14:54:50 +0000

⚠️Caution

Recently, managing EKS authentication via the API is preferred over using the aws-auth ConfigMap. Set authentication_mode to 'API' in the access configuration of the aws_eks_cluster resource, and streamline authentication management in Terraform by linking IAM roles with aws_eks_access_entry and aws_eks_access_policy_association.

REF

resource "aws_eks_cluster" "example" {
  name     = "example-cluster"
  ....
  access_config {
    authentication_mode                         = "API"
    bootstrap_cluster_creator_admin_permissions = false
  }
}

resource "aws_eks_access_entry" "example" {
  cluster_name      = aws_eks_cluster.example.name
  principal_arn     = aws_iam_role.example.arn
  kubernetes_groups = ["group-1", "group-2"]
  type              = "STANDARD"
}

resource "aws_eks_access_policy_association" "example" {
  cluster_name  = aws_eks_cluster.example.name
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
  principal_arn = aws_iam_user.example.arn

  access_scope {
    type       = "namespace"
    namespaces = ["example-namespace"]
  }
}

Introduction

Hi, everyone.
I would like to leave a memorandum about how to manage Kubernetes's configmap AWS auto-generated with terraform.

What Trouble

If we want to add iam user/role for eks cluster operation, we need to fix auto-generated aws-auth configmap(namespace:kube-system)
If we follow the official manual: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html), we should manage it with kubernetes manifest yml, but we want to manage it with terraform.
- Because at first we can access our eks cluster only with IAM user/role used when creating cluster(with ~/.kube/config as below) and our cluster generated role is terraform user/role
- Therefore, We want to add user/role to aws-auth configmap with terraform user/role and manage aws-auth configmap with terraform.
- Caution: According to EKS Best Practices Guides - Security, we should create the cluster with a dedicated IAM role

# ~/.kube/config
- name: arn:aws:eks:ap-northeast-1:9999999999:cluster/eks-example
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - ap-northeast-1
      - eks
      - get-token
      - --cluster-name
      - eks-example
      command: aws
      env: null
      provideClusterInfo: false

How to resolve the trouble

1. Add terraform aws-auth configmap resouece and Use `terraform import` command

1.0 Prepare terraform kubernetes provider

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks.token
}

Caution: we should set host/token/cluster_ca_certificate to operate kubernetes cluster with tf-provider.
- REF: https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest

1.1 Prapare aws-auth configmap tf resource for importing

# aws-auth.tf
resource "kubernetes_config_map" "aws-auth" {
  data = {
    "mapRoles" = ""
  }

  metadata {
    name      = ""
    namespace = ""
  }
}

1.2 Execute `terraoform import` cmd

terraform import kubernetes_config_map.aws-auth kube-system/aws-auth

1.3 terraform plan and remove diff from real resource state to resource config

resource "kubernetes_config_map" "aws-auth" {
  data = {
    "mapRoles" = <<EOT
- rolearn: arn:aws:iam::99999999999:role/hoge-role
  username: system:node:{{EC2PrivateDNSName}}
  groups:
    - system:bootstrappers
    - system:nodes
      # Therefore, before you specify rolearn, remove the path. For example, change arn:aws:iam::<123456789012>:role/<team>/<developers>/<eks-admin> to arn:aws:iam::<123456789012>:role/<eks-admin>. FYI:https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting_iam.html#security-iam-troubleshoot-ConfigMap
EOT
  }

  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }
}

2. Fix aws-auth configmap resource we imported and add iam user/role

resource "kubernetes_config_map" "aws-auth" {
  data = {
    "mapRoles" = <<EOT
- rolearn: arn:aws:iam::99999999999:role/hoge-role
  username: system:node:{{EC2PrivateDNSName}}
  groups:
    - system:bootstrappers
    - system:nodes
      # Therefore, before you specify rolearn, remove the path. For example, change arn:aws:iam::<123456789012>:role/<team>/<developers>/<eks-admin> to arn:aws:iam::<123456789012>:role/<eks-admin>. FYI:https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting_iam.html#security-iam-troubleshoot-ConfigMap
# Add as below 
- rolearn: hoge
  username: hoge
  groups: # REF: https://kubernetes.io/ja/docs/reference/access-authn-authz/rbac/
    - hoge
EOT
  }

  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }
}

DEV Community: Takashi Narikawa

My Python Project Development Guide ver.202412

What is this

Steps

1. Project Setup/ Set up Local Environment with uv

In the pyproject.toml file, we configure various modules for linting, formatting, and task running. Here's an explanation of these modules:

2. Create Directory Structure

3. Write Sufficient Unit Tests with pytest

4. Configure IDE Settings

5. Set up CI(Auto lint/test) with GitHub Actions

6. Set up Stage/Prod Environment with Dockerfile

7. Set up CD(Atuo build Dockerfile) with Github Actions

Appendix

Why are we organizing a tech conference called SRE NEXT 2022?

Introduction

What is SRE NEXT?

Why we hold SRE NEXT 2022

What kind of conference we want to hold

Conclusion

Introducing Conftest and setting up CI with Github Actions to automate reviewing of Terraform code

Introduction

Target readers

What is OPA/Rego/Conftest???

Write policy rules in Rego and set up CI with Github Actions

0.Prerequisites

0.1.Set up Conftest (OPA/Rego)

0.2.Structure of Terraform Plan results

1. Try to write Terrafom resource tag rules in Conftest/Rego

2. Write tests for the rule itself written in Rego with Conftest and try to run it.

3.Set CI to run Conftest with Github Action

3.1.Set up CI for fmt/verify of the policy rule itself written in Conftest/Rego.

3.2.Setting up CI to test Terraform plan results with Conftest test

Conclusion

References

Manage EKS aws-auth configmap with terraform

⚠️Caution

Introduction

What Trouble

How to resolve the trouble

1. Add terraform aws-auth configmap resouece and Use terraform import command

1.0 Prepare terraform kubernetes provider

1.1 Prapare aws-auth configmap tf resource for importing

1.2 Execute terraoform import cmd

1.3 terraform plan and remove diff from real resource state to resource config

2. Fix aws-auth configmap resource we imported and add iam user/role

References

1. Add terraform aws-auth configmap resouece and Use `terraform import` command

1.2 Execute `terraoform import` cmd