DEV Community: Spike

Define ObjectId for Pydantic 2.x compatible

Spike — Fri, 07 Jul 2023 12:06:52 +0000

I will update later.

For now, use this code.

Test Versions

FastAPI: 0.100.0
python: 3.11.4
pydantic: 2.0.2
pymongo: 4.4.0
pyhumps: 3.8.0

✨ Add new type PydanticObjectId

from collections.abc import Callable
from typing import Annotated, Any

import humps
from bson import ObjectId
from pydantic import BaseModel, ConfigDict, GetJsonSchemaHandler
from pydantic.json_schema import JsonSchemaValue
from pydantic_core import core_schema


class _ObjectIdPydanticAnnotation:
    @classmethod
    def __get_pydantic_core_schema__(
        cls,
        _source_type: Any,
        _handler: Callable[[Any], core_schema.CoreSchema],
    ) -> core_schema.CoreSchema:
        def validate_from_str(v: str) -> ObjectId:
            return ObjectId(v)

        from_str_schema = core_schema.chain_schema(
            [
                core_schema.str_schema(),
                core_schema.no_info_plain_validator_function(validate_from_str),
            ]
        )

        return core_schema.json_or_python_schema(
            json_schema=from_str_schema,
            python_schema=core_schema.union_schema(
                [
                    core_schema.is_instance_schema(ObjectId),
                    from_str_schema,
                ]
            ),
            serialization=core_schema.plain_serializer_function_ser_schema(lambda v: str(v)),
        )

    @classmethod
    def __get_pydantic_json_schema__(
        cls, _core_schema: core_schema.CoreSchema, handler: GetJsonSchemaHandler
    ) -> JsonSchemaValue:
        return handler(core_schema.str_schema())


PydanticObjectId = Annotated[ObjectId, _ObjectIdPydanticAnnotation]


class BaseODM(BaseModel):
    model_config = ConfigDict(
        alias_generator=humps.camelize,
        arbitrary_types_allowed=True,
        frozen=True,
        from_attributes=True,
        populate_by_name=True,
        str_strip_whitespace=True,
    )

    id: PydanticObjectId

👔 You don't have to convert data in FastAPI

With from_attributes=True, formerly known as orm_mode=True, you don't have to use the model_validate method.

import humps
from pydantic import BaseModel, ConfigDict


class EmbeddedBaseModel(BaseModel):
    model_config = ConfigDict(
        alias_generator=humps.camelize,
        arbitrary_types_allowed=True,
        frozen=True,
        from_attributes=True,  # 🪄 This is magic
        populate_by_name=True,
        str_strip_whitespace=True,
    )


class BaseODM(BaseModel):
    model_config = ConfigDict(
        alias_generator=humps.camelize,
        arbitrary_types_allowed=True,
        frozen=True,
        from_attributes=True,  # 🪄 This is magic
        populate_by_name=True,
        str_strip_whitespace=True,
    )

    id: PydanticObjectId

Reference

https://docs.pydantic.dev/latest/usage/types/custom/#handling-third-party-types

Migrate Terraform backend local to S3 and DynamoDB with multiple AWS accounts

Spike — Fri, 28 Apr 2023 08:21:46 +0000

Situation

You were the only developer in your company. You've managed AWS via Terraform and stored it in your local machine. As the company has grown up, you had to work with new colleagues. You needed to bring in a remote backend for several reasons; Collaboration, state management, disaster recovery, etc.

It's a widespread situation for early-stage startup companies. So, let's begin.

Multiple AWS accounts

Using multiple accounts often provides many benefits for managing AWS infrastructures. The one is dividing workloads into development and production. In addition, adding an account for common resources like secrets and config makes it better. There are lots of best practices, and you can see them from Organizing Your AWS Environment Using Multiple Accounts.

Prerequisite

I assume we have two OU(Organizational Units) and three accounts:

Fundamental(OU)
- Infrastructure (account_id: 999999999999)
Workloads(OU)
- development (account_id: 111111111111)
- production (account_id: 222222222222)

account_id is generated by AWS

It's not a recommendation but a fundamental OU architecture. We use AWS Identity Center, originally AWS SSO, to ease login to AWS Console and AWS CLI. This article is not about AWS Identity Center so we will skip it. But don't worry, if you are using IAM User, there is no problem to read.

Let's dive in

From now, assume we have terraform project like:

$ tree . -d
.
├── global
│   ├── 01-network
│   └── 20-ecs
└── services
    ├── service-a
    ├── service-b
    └── service-c

Create new terraform project

Create a directory, init, for S3 and DynamoDB and initializing it.

$ mkdir global/init
$ cd global/init
$ echo 'provider "aws" {}' > main.tf
$ terraform init

You can see the message like "Terraform has been successfully initialized".

Configure AWS SSO profiles

Create three profiles, dev, prod, infra via AWS CLI. See details in AWS CLI Configure SSO.

$ aws sso configure

You can see profiles via:

$ aws configure list-profiles
dev
prod
infra

Create S3 and DynamoDB

We are going to create S3 bucket and DynamoDB table in infra profile. Modify main.tf

# main.tf
provider "aws" {
  region  = "ap-northeast-2"  # rewrite your region
  profile = "infra"  # You must use `infra` profile which is in Fundamental OU
}

locals {
  # Use other ways to hide sensitive info e.g. `terraform.tfvars`
  principals = {
    dev  = "arn:aws:iam::111111111111:role/aws-reserved/sso.amazonaws.com/ap-northeast-2/AWSReservedSSO_AdministratorAccess_dev"
    prod = "arn:aws:iam::222222222222:role/aws-reserved/sso.amazonaws.com/ap-northeast-2/AWSReservedSSO_AdministratorAccess_prod"
    infra = "arn:aws:iam::999999999999:role/aws-reserved/sso.amazonaws.com/ap-northeast-2/AWSReservedSSO_AdministratorAccess_infra"
  }
}

resource "aws_s3_bucket" "tfstate" {
  bucket = "tfstate-bucket"
}

resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.tfstate.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_dynamodb_table" "tfstate_lock" {
  name         = "tfstate-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"  # Same as hash_key above
    type = "S"
  }
}

resource "aws_s3_bucket_policy" "this" {
  bucket = aws_s3_bucket.tfstate.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = "s3:ListBucket"
        Resource = "arn:aws:s3:::${aws_s3_bucket.tfstate.bucket}"
        Principal = {
          AWS = [for _, v in local.principals : v]
        }
      },
      {
        Sid      = "AllowAllS3ActionsInDev"
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]
        Resource = "arn:aws:s3:::${aws_s3_bucket.tfstate.bucket}/services/dev/*"
        Principal = {
          AWS = local.principals.dev
        }
      },
      {
        Sid      = "AllowAllS3ActionsInProd"
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]
        Resource = "arn:aws:s3:::${aws_s3_bucket.tfstate.bucket}/services/prod/*"
        Principal = {
          AWS = local.principals.prod
        }
      }
    ]
  })
}

We divided development and production into different paths.

arn:aws:s3:::${aws_s3_bucket.tfstate.bucket}/services/dev/*
arn:aws:s3:::${aws_s3_bucket.tfstate.bucket}/services/prod/*

Also, allow writing actions(s3:PutObjcet, etc.) for own paths. It prevents updating other tfstate which is not own.

Update legacy resource, `service-a`

Suppose we have an initialized terraform project and terraform file, services/service-a/versions.tf:

# services/service-a/versions.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.65.0"
    }
  }
}

provider "aws" {
  region = "ap-northeast-2"
}

It's an old one. Before modifying the file, we are going to create workspaces for SSOT. Create Terraform workspaces dev and prod.

$ terraform workspace new dev
$ terraform workspace new prod

You can see workspace list via:

$ terraform workspace list
  default
  dev
* prod

FYI, default workspace cannot be deleted.

Let's modify services/service-a/versions.tf

# services/service-a/versions.tf
terraform {
  backend "s3" {
    bucket               = "tfstate-bucket"
    workspace_key_prefix = "services"  # default is `env:`
    key                  = "service-a/terraform.tfstate"
    region               = "ap-northeast-2"

    profile = "infra"
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.65.0"
    }
  }
}

provider "aws" {
  region  = "ap-northeast-2"
  profile = terraform.workspace  # It should be one of `dev` and `prod`
}

Finally, migration

Enter terraform init in terminal.

$ terraform init

Initializing the backend...
Do you want to migrate all workspaces to "s3"?
  Both the existing "local" backend and the newly configured "s3" backend
  support workspaces. When migrating between backends, Terraform will copy
  all workspaces (with the same names). THIS WILL OVERWRITE any conflicting
  states in the destination.

  Terraform initialization doesn't currently migrate only select workspaces.
  If you want to migrate a select number of workspaces, you must manually
  pull and push those states.

  If you answer "yes", Terraform will migrate all states. If you answer
  "no", Terraform will abort.

  Enter a value:

Enter yes and you just have migrate remote backend to S3!🎉

If you want to apply other workload, you just change terraform workspace.

$ terraform workspace select dev

AWS ECS container health check

Spike — Fri, 25 Nov 2022 06:57:48 +0000

What is Container health check?

Container health check is a feature of Docker. It checks container is healthy or not(unhealthy). You don't have to use it, but it provides container status and is easy to check container status when entering docker ps command.

$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS                    PORTS                    NAMES
1b5c86ccfa6d   redis     "docker-entrypoint.s…"   13 seconds ago   Up 13 seconds (healthy)   0.0.0.0:6379->6379/tcp   redis

You can see health info from STATUS or use docker inspect

$ docker inspect redis | jq ".[0].State.Health"
{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2022-11-25T06:35:47.907103919Z",
      "End": "2022-11-25T06:35:47.973231669Z",
      "ExitCode": 0,
      "Output": "PONG\n"
    }
  ]
}

ExitCode is the result of the command in HEALTHCHECK instruction. If ExitCode is non-zero, container is unhealthy. I'll explain it in detail below.

How to use health check

It's straightforward. Use HEALTHCHECK instruction in Dockerfile.

HEALTHCHECK [OPTIONS] CMD command

Details in Dockerfile Docs.

FROM redis:7-alpine

HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
    CMD redis-cli ping

redis-cli ping does a health check. If you want to check the HTTP server, you can write

CMD-SHELL curl -f http://localhost:80/ || exit 1

instead. CMD-SHELL is similar to CMD, but the former uses the container's default shell.

CAUTION: You should install curl in the container when using curl command. If not, the container will be unhealthy on ECS even if there's no problem in the local environment. Redis in the above has redis-cli command already, so it's okay.

The options that can appear before CMD are:

--interval=DURATION default is 30s
--timeout=DURATION default is 30s
--start-period=DURATION default is 0s
--retries=N default is 3

ECS container health check

Okay, it's done, right? Unfortunately, ECS doesn't support container health check from Docker for monitoring.

AWS Docs says:

The Amazon ECS container agent only monitors and reports on the health checks specified in the task definition. Amazon ECS does not monitor Docker health checks that are embedded in a container image and not specified in the container definition. Health check parameters that are specified in a container definition override any Docker health checks that exist in the container image.

You have to put it into container definitions in task definition. Below is an incomplete ECS terraform example, so see only how it works.

# Three containers will be deployed: "app", "worker", "redis"
# Only "redis" has `healthCheck`
resource "aws_ecs_task_definition" "this" {
  family                   = "example"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 1024
  memory                   = 2048

  # other settings are omitted..

  container_definitions = jsonencode([
    {
      name      = "redis"
      image     = "redis:7-alpine"
      cpu       = 256
      memory    = 512
      essential = true

      portMappings = [{
        containerPort = 6379
        hostPort      = 6379
      }]

      healthCheck = {
        command     = ["CMD", "redis-cli", "ping"]
        interval    = 30
        retries     = 3
        startPeriod = 30
        timeout     = 5
      }
    },
    {
      name      = "app"
      image     = "django:latest"
      # other settings are omitted..
    },
    {
      name      = "worker"
      image     = "celery:latest"
      # other settings are omitted..
    }
  ])
}

After deploying the container, you can see redis container is Healthy, unlike the others are Unknown on AWS console.