AWS ECS container health check

#docker #ecs #aws #terraform

What is Container health check?

Container health check is a feature of Docker. It checks container is healthy or not(unhealthy). You don't have to use it, but it provides container status and is easy to check container status when entering docker ps command.

$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS                    PORTS                    NAMES
1b5c86ccfa6d   redis     "docker-entrypoint.s…"   13 seconds ago   Up 13 seconds (healthy)   0.0.0.0:6379->6379/tcp   redis

You can see health info from STATUS or use docker inspect

$ docker inspect redis | jq ".[0].State.Health"
{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2022-11-25T06:35:47.907103919Z",
      "End": "2022-11-25T06:35:47.973231669Z",
      "ExitCode": 0,
      "Output": "PONG\n"
    }
  ]
}

ExitCode is the result of the command in HEALTHCHECK instruction. If ExitCode is non-zero, container is unhealthy. I'll explain it in detail below.

How to use health check

It's straightforward. Use HEALTHCHECK instruction in Dockerfile.

HEALTHCHECK [OPTIONS] CMD command

Details in Dockerfile Docs.

FROM redis:7-alpine

HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
    CMD redis-cli ping

redis-cli ping does a health check. If you want to check the HTTP server, you can write

CMD-SHELL curl -f http://localhost:80/ || exit 1

instead. CMD-SHELL is similar to CMD, but the former uses the container's default shell.

CAUTION: You should install curl in the container when using curl command. If not, the container will be unhealthy on ECS even if there's no problem in the local environment. Redis in the above has redis-cli command already, so it's okay.

The options that can appear before CMD are:

--interval=DURATION default is 30s
--timeout=DURATION default is 30s
--start-period=DURATION default is 0s
--retries=N default is 3

ECS container health check

Okay, it's done, right? Unfortunately, ECS doesn't support container health check from Docker for monitoring.

AWS Docs says:

The Amazon ECS container agent only monitors and reports on the health checks specified in the task definition. Amazon ECS does not monitor Docker health checks that are embedded in a container image and not specified in the container definition. Health check parameters that are specified in a container definition override any Docker health checks that exist in the container image.

You have to put it into container definitions in task definition. Below is an incomplete ECS terraform example, so see only how it works.

# Three containers will be deployed: "app", "worker", "redis"
# Only "redis" has `healthCheck`
resource "aws_ecs_task_definition" "this" {
  family                   = "example"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 1024
  memory                   = 2048

  # other settings are omitted..

  container_definitions = jsonencode([
    {
      name      = "redis"
      image     = "redis:7-alpine"
      cpu       = 256
      memory    = 512
      essential = true

      portMappings = [{
        containerPort = 6379
        hostPort      = 6379
      }]

      healthCheck = {
        command     = ["CMD", "redis-cli", "ping"]
        interval    = 30
        retries     = 3
        startPeriod = 30
        timeout     = 5
      }
    },
    {
      name      = "app"
      image     = "django:latest"
      # other settings are omitted..
    },
    {
      name      = "worker"
      image     = "celery:latest"
      # other settings are omitted..
    }
  ])
}