DEV Community

Tetsuya KIKUCHI
Tetsuya KIKUCHI

Posted on

ECS External Deployment & TaskSet - Complete Guide

#aws #ecs #pipecd

This guide explores "External Deployment" and "TaskSet" - two of the lesser-known yet powerful features in ECS. We'll cover their concepts and practical usage, including real-world examples with PipeCD. Mastering these features enables advanced deployment strategies in ECS.

Note: This article is based on specifications as of December 14, 2024.

Key Takeaways

  • TaskSet: A layer between Service and Task that maintains revisions per TaskSet
  • External Deployment: A deployment type that enables flexible deployments using TaskSets
  • These features are essential for implementing advanced deployment strategies like Canary in ECS

Understanding the Terminology

1. What is External Deployment?

External Deployment is a deployment type in ECS that enables flexible deployments through third-party controllers.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-external.html

Note: This is unrelated to launchType: EXTERNAL used in ECS Anywhere.

Three Deployment Types

ECS offers three deployment types, with External Deployment being one of them:

  • ECS: Rolling update - the default and most conventional approach
  • CODE_DEPLOY: Integrates with CodeDeploy for straightforward Canary and Blue/Green deployments
  • EXTERNAL: Enables custom deployments through TaskSet manipulation

Important: The deployment type cannot be changed after service creation.

Differences between CODE_DEPLOY and EXTERNAL

  • Both manipulate TaskSets and follow similar operational flows[^1]
  • CODE_DEPLOY is easier to implement as CodeDeploy handles most operations
  • EXTERNAL offers more flexibility despite its complexity
    • Examples: Custom deployment validation, rollback strategies

2. What is TaskSet?

TaskSet is a concept that sits between ECS Service and Task, grouping Tasks together.

TaskSets only appear when using CodeDeploy or External deployment types, handling Task scheduling and scaling instead of the Service.

You won't encounter TaskSets when using rolling updates or running Standalone Tasks.

Examining TaskSets

TaskSets aren't visible in the console. You can view them by using DescribeServices on a Service using CodeDeploy or External deployment. They have similar attributes to Services, including runningCount, launchType, and networkConfiguration.

$ aws ecs describe-services --cluster xxx --services yyy-service
{
    "services": [
        {
            "serviceName": "yyy-service",
            "taskSets": [ // Here
                {
                    "id": "ecs-svc/4390851359570905103",
                    "taskSetArn": "arn:aws:ecs:ap-northeast-1:<account>:task-set/xxx/yyy-service/ecs-svc/4390851359570905103",
                    "serviceArn": "arn:aws:ecs:ap-northeast-1:<account>:service/xxx/yyy-service",
                    "clusterArn": "arn:aws:ecs:ap-northeast-1:<account>:cluster/xxx",
                    "status": "ACTIVE",
                    "taskDefinition": "arn:aws:ecs:ap-northeast-1:<account>:task-definition/zzz-taskdef:1",
                    "computedDesiredCount": 2,
                    "pendingCount": 2,
                    "runningCount": 0,
                    "createdAt": "2024-12-13T20:00:24.064000+09:00",
                    "updatedAt": "2024-12-13T20:00:28.178000+09:00",
                    "launchType": "FARGATE",
                    "platformVersion": "1.4.0",
                    "platformFamily": "Linux",
                    "networkConfiguration": {
                        "awsvpcConfiguration": {
                            "subnets": [
                                "subnet-aaa",
                                "subnet-bbb"
                            ],
                            "securityGroups": [
                                "sg-ccc"
                            ],
                            "assignPublicIp": "DISABLED"
                        }
                    },
                    "loadBalancers": [],
                    "serviceRegistries": [],
                    "scale": {
                        "value": 100.0,
                        "unit": "PERCENT"
                    },
                    "stabilityStatus": "STABILIZING",
                    "stabilityStatusAt": "2024-12-13T20:00:24.064000+09:00",
                    "tags": []
                }
            ],
            "deploymentController": {
                "type": "EXTERNAL" // External deployment
            },
            ...
        }
    ],
}
Enter fullscreen mode Exit fullscreen mode

When to Use TaskSets

TaskSets enable Canary releases and Blue/Green deployments because you can specify different TargetGroups and TaskDefinitions per TaskSet:

  • "TaskSet with v1 TaskDefinition points to TargetGroup A"
  • "TaskSet with v2 TaskDefinition points to TargetGroup B"

Control traffic distribution between TargetGroups at the ALB level.


Canary release and Blue/Green deployment using TaskSets

TaskSet Operations

There are only five APIs for TaskSet operations. Most operations involve managing multiple TaskSets rather than modifying a single TaskSet's internals.

  1. CreateTaskSet
  2. DescribeTaskSets
    • Requires TaskSet ID beforehand, making it cumbersome
    • DescribeServices is recommended for easier TaskSet information retrieval
  3. DeleteTaskSet
  4. UpdateTaskSet
    • Can only update scale
      • scale: Percentage of Tasks to run in this TaskSet relative to Service's desiredCount
      • Example: With desiredCount 10 and scale 30%, TaskSet maintains 3 Tasks
      • Range: 0-100
      • Task count automatically adjusts when Service autoscales (changes in desiredCount)
    • Cannot update taskDefinition, ensuring all Tasks within a TaskSet use the same image with immutable tags
  5. UpdateServicePrimaryTaskSet
    • Changes a TaskSet's status to PRIMARY
    • Optional but recommended as Primary TaskSet cannot be deleted with DeleteTaskSet1
    • Making a TaskSet Primary changes the previous Primary TaskSet to ACTIVE
    • Some Primary TaskSet settings propagate to Service (launchType, taskDefinition, etc.)
 - Before Primary TaskSet exists:
Enter fullscreen mode Exit fullscreen mode 
     ```json
     $ aws ecs describe-services --cluster xxx --services yyy-service
     {
         "services": [
             {
                 "serviceName": "yyy-service",
                 "launchType": "EC2", // Initial Service creation
                 "taskSets": [
                     {
                         "id": "ecs-svc/8721983045402263235",
                         "status": "ACTIVE", // Not PRIMARY yet
                         "taskDefinition": "arn:aws:ecs:ap-northeast-1:<account>:task-definition/zzz-taskdef:3",
                         "launchType": "FARGATE",
                         ...
     ```
Enter fullscreen mode Exit fullscreen mode 
 - After `UpdateServicePrimaryTaskSet`:
Enter fullscreen mode Exit fullscreen mode 
     ```json
     $ aws ecs describe-services --cluster xxx --services yyy-service
     {
         "services": [
             {
                 "serviceName": "yyy-service",
                 "launchType": "FARGATE", // Propagated from TaskSet
                 "taskDefinition": "arn:aws:ecs:ap-northeast-1:<account>:task-definition/zzz-taskdef:3", // Propagated from TaskSet
                 "taskSets": [
                     {
                         "id": "ecs-svc/8721983045402263235",
                         "status": "PRIMARY", // Now PRIMARY
                         "taskDefinition": "arn:aws:ecs:ap-northeast-1:<account>:task-definition/zzz-taskdef:3",
                         "launchType": "FARGATE",
                         ...
     ```
Enter fullscreen mode Exit fullscreen mode

Example: Canary Release with External Deployment

There are multiple approaches to External Deployment, including rollback strategies.
Here are two examples.
For more details, refer to this documentation.

Note: This article focuses on Services behind ALB.

Method A: Promoting Canary to Primary

Prerequisites

Assume ALB, ListenerRule, two TargetGroups, ECS Cluster, and TaskDefinitions (v1,v2) are ready

Step 1: Create Service

Use CreateService to create a Service.

  • [Required] Set deploymentController.type to EXTERNAL

  • Fewer required parameters compared to ECS deployment type

    • No need for taskDefinition, launchType, networkConfiguration
    • Details here 
$ aws ecs create-service \
--cluster <cluster-name> \
--service-name <service-name> \
--desired-count 2 \
--deployment-controller type=EXTERNAL
Enter fullscreen mode Exit fullscreen mode

No TaskSets or Tasks are created at this point.

Step 2: Create Initial TaskSet (v1)

Use CreateTaskSet to create the first TaskSet.

  • Specify parameters typically set at Service level (launchType, taskDefinition, etc.)
  • loadBalancers[].targetGroupArn: Specify TargetGroup for ALB traffic
  • Recommend 100% scale for initial TaskSet 
$ aws ecs create-task-set \
--cluster <cluster-name> \
--service <service-name> \
--task-definition <task-def-arn-v1> \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[subnet-aaa,subnet-bbb],securityGroups=[sg-ccc],assignPublicIp=ENABLED}" \
--load-balancers "targetGroupArn=<tg-arn-1>,containerName=web,containerPort=80" \ # Match TaskDefinition
--scale unit=PERCENT,value=100
Enter fullscreen mode Exit fullscreen mode


After prerequisites

Note: UpdateServicePrimaryTask is optional here

Deployment Flow

1. Create TaskSet (v2)

Similar to Step 2 above, but with different task-definition, targetGroupArn, and scale.

$ aws ecs create-task-set \
--cluster <cluster-name> \
--service <service-name> \
--task-definition <task-def-arn-v2> \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[subnet-aaa,subnet-bbb],securityGroups=[sg-ccc],assignPublicIp=ENABLED}" \
--load-balancers "targetGroupArn=<tg-arn-2>,containerName=web,containerPort=80" \ # Match TaskDefinition
--scale unit=PERCENT,value=10
Enter fullscreen mode Exit fullscreen mode

No traffic flows to v2 TaskSet yet as its TargetGroup weight is 0.


Creating Canary TaskSet

2. Route Traffic to Canary

Use ELB's ModifyRule to adjust TargetGroup weights.

$ aws elbv2 modify-rule \
--rule-arn <listener-rule-arn> \
--actions '[{
    "Type": "forward", 
    "ForwardConfig": {
        "TargetGroups": [
            {
                "TargetGroupArn": "<tg-arn-1>", 
                "Weight": 90
            },
            {
                "TargetGroupArn": "<tg-arn-2>", 
                "Weight": 10
            }
        ]
    }
}]'
Enter fullscreen mode Exit fullscreen mode


Starting traffic flow to Canary

Use ModifyListener for default listener rules.
For multiple ListenerRules, gather all target rules beforehand.

3. If Successful, Route 100% Traffic to Canary

Note: If Canary's scale was less than 100, first use UpdateTaskSet to set it to 100.

Use ELB's ModifyRule again to update TargetGroup weights.

$ aws elbv2 modify-rule \
--rule-arn <listener-rule-arn> \
--actions '[{
    "Type": "forward", 
    "ForwardConfig": {
        "TargetGroups": [
            {
                "TargetGroupArn": "<tg-arn-1>", 
                "Weight": 0
            },
            {
                "TargetGroupArn": "<tg-arn-2>", 
                "Weight": 100
            }
        ]
    }
}]'
Enter fullscreen mode Exit fullscreen mode


Routing 100% traffic to Canary

4. Promote Canary TaskSet to Primary

4.1. Get Canary TaskSet ID 

$ aws ecs describe-services \
--cluster <cluster-name> \
--services <service-name> \
| jq '.services[0].taskSets[]'
Enter fullscreen mode Exit fullscreen mode

Find Canary TaskSet ID in response, using status (PRIMARY/ACTIVE) to identify it.

4.2. Use UpdateServicePrimaryTaskSet to Make Canary TaskSet Primary 

$ aws ecs update-service-primary-task-set \
--cluster <cluster-name> \
--service <service-name> \
--primary-task-set <ecs-svc/xxx> # Canary TaskSet ID from 4.1
Enter fullscreen mode Exit fullscreen mode

5. Delete Old TaskSet

Use DeleteTaskSet to remove v1 TaskSet.

$ aws ecs delete-task-set \
--cluster <cluster-name> \
--service <service-name> \
--task-set <ecs-svc/xxx> # Old TaskSet ID from 4.1
Enter fullscreen mode Exit fullscreen mode


After deployment completion

Rollback Scenarios

  • If failure occurs during steps 2 or 3:

    1. Return 100% traffic to TargetGroup A
    2. Delete Canary TaskSet

  • If failure occurs during step 5:

    1. Create new TaskSet using TaskDefinition v1 and TargetGroup A
    2. Update it to Primary
    3. Route 100% traffic to TargetGroup A
    4. Delete Canary TaskSet

Method A Challenges

TargetGroup assignments to old/new TaskSets aren't fixed, requiring tracking of current assignments.
This complicates rollback especially.

Method B: Using Three TaskSets

Here's an alternative approach that addresses Method A's challenges.
It uses three TaskSets: "current", "canary", and "new".

This is the approach PipeCD adopts.

Deployment Flow

0. Initial State

1. Create Canary

Use CreateTaskSet and elbv2::ModifyRule.
Canary TaskSet's scale can be less than 100%.

2. If Successful, Create New TaskSet

Use CreateTaskSet with TargetGroup A and TaskDefinition v2.

Traffic now flows to the third TaskSet.

3. Delete All Old TaskSets

Use UpdateServicePrimaryTaskSet, elbv2::ModifyRule, and DeleteTaskSet.

Back to initial state configuration.
Next deployment starts with TargetGroup A pointing to latest version.

Rollback Scenarios

  • For any failure phase, return to initial state:
  1. Create new TaskSet with TaskDefinition v1 and TargetGroup A
  2. Update it to Primary
  3. Route 100% traffic to TargetGroup A
  4. Delete all other TaskSets
  • For failures before step 3, faster rollback:
  1. Route 100% traffic to TargetGroup A
  2. Delete all other TaskSets

Pros/Cons

Pros:

  • Fixed TargetGroup assignments simplify operations

Cons:

  • Longer deployment time due to two TaskSet creations
  • Higher cost with three TaskSets

Works on EC2 Too

Method B works on EC2 as well as Fargate.

  • Initially concerned about deploymentConfiguration.maximumPercent limitations
  • Worried about "Can't use Method B if limited to 2x desiredCount Tasks", but maximumPercent is ignored
  • Note: Docs explicitly state that maximumPercent is ignored for External Deployment on Fargate

Using Method B with PipeCD

External Deployment can be complex (especially with rollbacks).
PipeCD implements Method B Canary releases with simple pipeline definitions.
Handles rollbacks automatically.

  pipeline:
    stages:
      # 1. Create Canary TaskSet
      - name: ECS_CANARY_ROLLOUT 
        with:
          scale: 10
      # 2. Route 10% traffic to Canary
      - name: ECS_TRAFFIC_ROUTING 
        with:
          canary: 10
      # 3. Approval phase
      - name: WAIT_APPROVAL
      # 4. Create new Primary TaskSet
      - name: ECS_PRIMARY_ROLLOUT 
      # 5. Route 100% traffic to new Primary TaskSet
      - name: ECS_TRAFFIC_ROUTING 
        with:
          primary: 100
       # 6. Delete unnecessary TaskSets
      - name: ECS_CANARY_CLEAN
Enter fullscreen mode Exit fullscreen mode

Full configuration file here.

Important Considerations for External Deployment

Several important points to consider when using External Deployment.

Many Service Features/Settings Unsupported

  • Examples of unsupported features:

  • No comprehensive list of unsupported features exists

    • Recommend searching docs for keywords like "Deployment"
    • Service Connect example: > Only services that use rolling deployments are supported with Service Connect.

  • No known TaskDefinition-specific constraints for External Deployment

External Deployment Often Excluded from ECS Updates

Migration from Other Deployment Types is Challenging

  • As mentioned, deployment type can't be changed after Service creation (particularly painful)
    • Migrating to External Deployment requires creating new Service and gradually shifting traffic
  • Many unsupported features mean reviewing Service settings
    • Some settings move from CreateService to CreateTaskSet

Manual TaskSet Tagging Required

Limited Console Information

  • TaskSets not visible in ECS console, requiring CLI usage

  • "Deployment History" remains unchanged for External Deployment

  • Reference: PipeCD visualizes Service/TaskSet/Task states in UI 2

    PipeCD UI (during Canary deployment)

Other Constraints

  • Can't link multiple TargetGroups to one TaskSet

  • Maximum 5 TaskSets per Service

    • Allows up to 5x desiredCount Tasks per Service (scale100 x 5 TaskSets)
    • Not documented
    • Attempting to exceed limit results in: 
    An error occurred (InvalidParameterException) when calling the CreateTaskSet operation: 5 deployments exist on the service. Unable to create new TaskSet since this exceeds the maximum limit.

Additional Notes

Note 1: Why Low Awareness?

Two hypotheses for External Deployment and TaskSet's low recognition:

  1. Minimal console presence

    Service creation screen - "External Deployment" not visible

  2. Satisfaction with ECS/CODE_DEPLOY deployment types (but are they truly sufficient?)

Note 2: Do TaskSets Exist in ECS Deployment Type?

A: Possibly used internally, but unclear

Two hints suggest TaskSet-like behavior:

  1. Task's StartedBy format matches External Deployment's ecs-svc/xxx
    • TaskSet IDs use ecs-svc/xxx format, and Tasks within TaskSets show this as StartedBy
  2. The xxx in ecs-svc/xxx matches service revision ID

Service revision mechanism seems similar to TaskSets but has different ARN format.

DescribeTaskSets on ecs-svc/xxx fails:

$ aws ecs describe-task-sets --cluster aaa --service bbb --task-sets ecs-svc/xxx

An error occurred (InvalidParameterException) when calling the DescribeTaskSets operation: Amazon ECS only supports task set management on services configured to use external deployment controllers.
Enter fullscreen mode Exit fullscreen mode

Conclusion

External Deployment, while sparsely documented and constrained, is powerful when mastered.
Service Connect and VPC Lattice support would be welcome additions.

While migrating to External Deployment is challenging, PipeCD will soon support other deployment types through "plugins".
For plugin details, see:

https://pipecd.dev/blog/2024/11/28/overview-of-the-plan-for-pluginnable-pipecd/

  1. DeleteTaskSet will return this error: An error occurred (InvalidParameterException) when calling the DeleteTaskSet operation: Primary TaskSet cannot be deleted ↩

  2. https://pipecd.dev/docs/user-guide/managing-application/application-live-state/ ↩

Retry later

Top comments (1)

Collapse
 
t-kikuc profile image
Tetsuya KIKUCHI •

I translated from my article: zenn.dev/cadp/articles/ecs-external