DEV Community: Tetsuya KIKUCHI

ECS Native Blue/Green is Here! With Strong Hooks and Dark Canary

Tetsuya KIKUCHI — Sun, 20 Jul 2025 05:42:48 +0000

On July 18, 2025, Amazon ECS received a major deployment enhancement. It's not just about native Blue/Green support - there's much more to it!

This article is translated from my article in Japanese.

Key Points

Native Blue/Green is now available without CodeDeploy
- Various validation timings through lifecycle hooks with Lambda
- Pre-validation in production environment with zero user impact (Dark Canary) using test listeners/listener rules
Blue/Green is now supported with Service Connect
Deployment controller can be changed after service creation
You should avoid CodeDeploy-based Blue/Green (migration guide available)

Note: This article does not cover Blue/Green with Service Connect.

Update Overview

https://aws.amazon.com/about-aws/whats-new/2025/07/amazon-ecs-built-in-blue-green-deployments/

Blue/Green deployment is now available as a built-in ECS feature without requiring CodeDeploy.
You can select it directly from the console:

This B/G deployment comes with two optional features that make automated and safe deployments easier:

Deployment lifecycle hooks for custom validation
Test listener / listener rules for Dark Canary

Also, there's a subtle but significant update: "deployment controller can be changed after service creation."

Previous Situation

When implementing Blue/Green in ECS, combining with CodeDeploy was common but had several pain points:

Required cumbersome CodeDeploy setup
Couldn't switch between rolling update ↔ B/G after service creation
Had various constraints when combined with CodeDeploy
- Example: Couldn't use Service Connect

Rolling updates also had challenges:

Limited flexibility in success/failure determination (despite having CloudWatch alarms and deployment circuit breakers)
Time-consuming rollbacks (requiring new task launches)

Benefits of This Update

The update brings several significant improvements:

Easy Blue/Green deployment without CodeDeploy setup.
Able to switch between rolling update and B/G even after service creation.
- Basically just change the strategy. No service recreation & migration needed.
- Easy to "start simple with rolling update, switch to B/G when needed".
- Detailed migration guide in this documentation.
- Also includes B/G → rolling update migration, suggesting rolling updates aren't deprecated.
Features from CodeDeploy are available, making it convenient + easy to migrate from CodeDeploy.
- Flexible validation with Lambda.
- Zero-impact new version testing in production environment.
Being a native feature, likely to have fewer constraints than CodeDeploy integration.
Blue/Green deployment now works with Service Connect.
- Removes one drawback of Service Connect and shows it's being actively maintained.

Detailed Features

Deployment Lifecycle Hooks

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-lifecycle-hooks.html

In native Blue/Green, you can validate deployment success using custom logic through Lambda functions at various stages.
For example, you can monitor service status, access endpoints, or check telemetry data.

This is similar to CodeDeploy's hooks feature.

Lifecycle Stages

There are 7 hook timings (lifecycle stages):

PRE_SCALE_UP: Before new tasks launch
POST_SCALE_UP: After new tasks launch and become healthy
TEST_TRAFFIC_SHIFT: During test traffic shift to Green (0->100%)
POST_TEST_TRAFFIC_SHIFT: After test traffic is 100% on Green
PRODUCTION_TRAFFIC_SHIFT: During production traffic shift to Green
POST_PRODUCTION_TRAFFIC_SHIFT: After production traffic shift to Green
RECONCILE_SERVICE: When deployment starts with multiple ACTIVE service revisions
- Not selectable in console but available via CLI. Purpose unclear.

During rollback, TEST_TRAFFIC_SHIFT and PRODUCTION_TRAFFIC_SHIFT are hooked.

Event Payload

The event payload includes service ARN and weight information, allowing validation logic based on these values.

Example:

Event: 
{
    "executionDetails": {
        "testTrafficWeights": {},
        "productionTrafficWeights": {
            "arn:aws:ecs:ap-northeast-1:<account-id>:service-revision/my-cluster/native-bg-1/9942985458929989075": 0,
            "arn:aws:ecs:ap-northeast-1:<account-id>:service-revision/my-cluster/native-bg-1/2948000638822554633": 100
        },
        "serviceArn": "arn:aws:ecs:ap-northeast-1:<account-id>:service/my-cluster/native-bg-1",
        "targetServiceRevisionArn": "arn:aws:ecs:ap-northeast-1:<account-id>:service-revision/my-cluster/native-bg-1/2948000638822554633"
    },
    "executionId": "06a4bc13-a7fa-4281-ab04-3aa34234ddxx",
    "lifecycleStage": "PRODUCTION_TRAFFIC_SHIFT",
    "resourceArn": "arn:aws:ecs:ap-northeast-1:<account-id>:service-deployment/my-cluster/native-bg-1/PNpQryOI09kD3iMrxsoxx"
}

Function Return Values

hookStatus=SUCCEEDED: Validation successful, deployment proceeds
hookStatus=FAILED: Triggers rollback
hookStatus=IN_PROGRESS: Function called again after a delay
- Useful for long-running checks or when validation data isn't yet available
- Official blog mentions 30-second intervals, confirmed in testing

Note: Partially Available in Rolling Update??

While the console doesn't show lifecycle hooks or bake time settings for rolling updates,
CLI allows selecting these with rolling updates. LB settings from B/G remain.

In actual deployment, only the PRE_SCALE_UP hook triggered Lambda. Unclear if this is intended behavior.

Test Listener / Listener Rule (Dark Canary)

Using test listeners/listener rules, developers/testers can access the Green environment before production traffic shifts.
This is called "Dark Canary" as end users don't access it.

Benefits

Compared to simple Blue/Green, this reduces the risks of:

Complete disaster when 100% traffic is shifted to Green, even temporarily
"Works in staging but fails in production" scenarios

Usage

Create separate access routes for developers using:

Listeners with different ports
Listener rules with conditions (headers, source IPs, etc.)

This phase is validated by TEST_TRAFFIC_SHIFT and POST_TEST_TRAFFIC_SHIFT hooks.

Return hookStatus=IN_PROGRESS for zero-impact rollback
Deployment stays IN_PROGRESS while returning hookStatus=IN_PROGRESS (timeout unknown, confirmed >3 hours)
- For manual validation, consider having Lambda monitor a flag and return hookStatus=SUCCEEDED when set

Additionally, Deployment Controller Now Updatable Post-Creation

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-parameters.html

A subtle but important documentation update.

Background: There are 3 deployment controllers:

ECS (Enhanced now, most common)
CODE_DEPLOY (Traditional B/G deployment)
EXTERNAL (For customization. Details: ECS External Deployment & TaskSet Guide)

Previously unchangeable after service creation, now supports 4 update patterns:

CODE_DEPLOY -> ECS
CODE_DEPLOY -> EXTERNAL
ECS -> EXTERNAL
EXTERNAL -> ECS

Hmm?

Signs of `CODE_DEPLOY` Type Deprecation Not CodeDeploy itself

Notice no update patterns TO CODE_DEPLOY.

The CODE_DEPLOY docs clearly recommend the new native B/G:

We recommend that you use the Amazon ECS blue/green deployment.

CODE_DEPLOY option removed from console.
Migration docs provided:

This likely prompted deployment controller update support.

No deprecation notices or migration guides for EXTERNAL, suggesting it's safe.

Like EKS on Fargate Auto Mode, nice to see deprecation/removal after superior alternatives emerge. Unlike certain other cases...

Benefits of This Update

Makes CODE_DEPLOY to ECS migration easier.

Also greatly simplifies PipeCD migration for ECS.

PipeCD uses EXTERNAL for ECS deployments.
Previously, migrating from rolling update(ECS) or CODE_DEPLOY required service recreation.
For running services, complex ALB listener-based migration was needed.

Now possible without service recreation (both to and from PipeCD).

Also enables:

"Switch from ECS to EXTERNAL for customization"
"Try EXTERNAL, revert to ECS if too complex"

Note

Can't migrate from ECS if using VPC Lattice or Service Connect:

You can't update the deployment controller of a service from the ECS deployment controller to any of the other controllers if it uses VPC Lattice or Amazon ECS Service Connect.

How It Works (ALB Case)

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bluegreen-how-it-works.html

Deployment Flow

Created diagram as official one felt incomplete:

Initial State ~ Green Environment Launch
1. Initial state: Blue environment receiving 100% traffic
2. Launch Green tasks, attach to Green target group
3. ALB health checks Green environment
Internal Green environment testing (no production traffic)
Switch production traffic to Green
- Brief for "All at once"
Post-Green Switch ~ Deployment Complete
1. Monitor: Watch CloudWatch alarms, auto-rollback if issues
  - Continues until Bake Time parameter expires
2. Delete Blue tasks
3. Deployment complete

Next deployment reverses Blue/Green, moving from Target Group Green to Blue.

During Rollback

Rollback simply returns traffic to coexisting Blue environment via listener rules.
Faster than rolling update as no task launches needed.

Hands-On Testing

Following this official blog:

https://aws.amazon.com/blogs/aws/accelerate-safe-software-releases-with-new-built-in-blue-green-deployments-in-amazon-ecs/

Resource details here:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/alb-resources-for-blue-green.html

1. Service Update

Configuration:

task definition: httpd → nginx

Deployment options

Deployment controller type: ECS
Previously chose between ECS or CODE_DEPLOY
strategy: Blue/Green
Bake time: 5 minutes
lifecycle hooks:
Lambda function: Simple function returning "hookStatus": "SUCCEEDED" after accessing ALB URL

  import json
  import urllib3
  import logging
  import base64
  import os

  # Configure logging
  logger = logging.getLogger()
  logger.setLevel(logging.DEBUG)

  # Initialize HTTP client
  http = urllib3.PoolManager()

  def lambda_handler(event, context):
      """
     Validation hook that tests the green environment by accessing "/"
      """
     logger.info(f"Event: {json.dumps(event)}")
     logger.info(f"Context: {context}")

      try:
     test_endpoint = os.getenv("APP_URL")

     response = http.request(
              'GET', 
     test_endpoint,
              timeout=30
          )

     logger.info(f"GET / response status: {response.status}")

          # Check if response has OK status code (200-299 range)
          if 200 <= response.status < 300:
     logger.info("test passed - received OK status code")
              return {
                  "hookStatus": "SUCCEEDED"
              }
          else:
     logger.error(f"test failed - status code: {response.status}")
              return {
                  "hookStatus": "FAILED"
              }

      except Exception as error:
     logger.error(f"test failed: {str(error)}")
          return {
              "hookStatus": "FAILED"
          }

Role: Role with lambda:InvokeFunction. Reference
- Role for ECS to invoke Lambda
Lifecycle stages: All 6 selected

Load balancing
- Role: Policy based on this doc
- Role for ECS to update listener rules
- Added elasticloadbalancing permissions (DescribeTargetGroups, DescribeTargetHealth,RegisterTargets,DeregisterTargets) due to permission errors
- Load balancer type: ALB
- Listener (production): HTTP:80
- Production listener rule: Listener default
- Test listener (Green test access): Different port HTTP:81
- Test listener rule: Listener default
- Target group (Blue): IP type with HTTP:80
- Alternate target group (Green): Same settings as Blue
- "Create alternate target group" option creates with just naming

2. Deployment

2-1. Test Traffic

First, Green tasks launched and POST_SCALE_UP lifecycle hooks succeeded.
Green accessible via test listener (HTTP:81).

Green port 81 showed nginx:

Blue port 80 showed httpd:

ALB test listener (port 81) rule changed to Green (group2):

Production listener (port 80) rule still Blue:

POST_TEST_TRAFFIC_SHIFT lifecycle hooks succeeded.

2-2. Production Traffic Switch

Production traffic switches to Green.

Port 80 access switched to nginx:

ALB production listener rule switched to Green:

Test listener still accessed Green environment.

2-3. Bake Time

Blue(Source) tasks still running for fast rollback, no production traffic:

After bake time, Blue tasks deleted:

Event history shows:

Deployment Status Monitoring

Current stage visible in Deployments screen:

Click stage to see hook-Lambda function mappings:

Personally, stage start/end status in Events would help track timing and troubleshoot.

3. Testing Hook Failure

Testing rollback by failing POST_SCALE_UP lifecycle hooks.

1. Replace Lambda function

Use this always failing function:

import logging
import json

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

def lambda_handler(event, context):
 logger.info(f"always return failure")
    return {
        "hookStatus": "FAILED"
    }

2. Change Lifecycle hooks to POST_SCALE_UP only

Avoid "Rollback failed" from hook failures during rollback's PRODUCTION_TRAFFIC_SHIFT.

3. Update service to trigger deployment

Changed task definition revision.

POST_SCALE_UP failed, triggering rollback:

Notes

No Canary support
- CodeDeploy had Canary option, hopefully added later
- Traffic shifting section looks ready for options...
Auto Scaling warning:

If your service uses auto scaling, be aware that auto scaling is not blocked during a blue/green deployment, but the deployment might fail under certain circumstances.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-bluegreen.html

Note: "Rolling Update" Deployment Type Naming Issue

What should we call deployment type ECS now?
Previously "ECS (rolling update)", but now includes B/G.

Documentation still refers to "using rolling update" likely meaning deployment type=ECS. Awaiting updates. Example:

Only services that use rolling deployments are supported with Service Connect.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect-concepts-deploy.html#service-connect-considerations

Conclusion

This is one of ECS's most significant updates recently, including changing deployment controller.
I'm curious about Service Connect's Blue/Green implementation.

I'm relieved about the continued support for Service Connect and External deployment. However, the CODE_DEPLOY type should probably be avoided going forward.

Comparing 5 Methods of ECS Interservice Communication Including VPC Lattice

Tetsuya KIKUCHI — Tue, 22 Apr 2025 04:46:13 +0000

This article is translated from my original article in Japanese.

Introduction

This article compares five options for ECS service-to-service communication: ALB, VPC Lattice, ECS Service Discovery, ECS Service Connect, and App Mesh (scheduled for deprecation).

Background

Following the App Mesh deprecation announcement and VPC Lattice enhancements at re:Invent 2024 (ECS native integration, TCP support), I wanted to provide an overview of the future landscape of ECS interservice communication.

This article is based on specifications as of December 18, 2024.

Summary

	ALB	VPC Lattice	ECS Service Discovery	ECS Service Connect	App Mesh (Deprecated)
In short	Stable & versatile	Evolution of ALB that connects everything	Simple name resolution	ECS-specific, easy & feature-rich	Complex & feature-rich mesh
Release (GA)	2016	2023	2018	2022	2019
Ease of setup & management	Medium	Medium	👍👍Easy	👍Relatively easy	👿👿Complex
💰Cost	Time + LCU	Time + traffic + request count	👍👍Very cost-effective	Sidecar	Sidecar
🔭Logs & Metrics	👍Supported	👍Supported	👿None	👍Supported	👍Supported
🔭Tracing	👿None	👿None	👿None	👿None	👍Supported
🛡️Health Check	👍Supported	👍Supported	👍Supported (container level)	👍Partially supported¹	👍Supported
🛡️Auto Retry	👿None	👿None	👿None	👍Supported	👍Supported
🛡️Circuit Breaker	👿None	👿None	👿None	Partial (outlier detection)	👍Supported
🚢Supported Deployment Types	👍All	Technically all	👍All	👿`ECS` only	👍All
🚢Canary Deployment	👍Easy	👍Easy	👿Tricky	👿Not possible	👍Easy
🐙Name Resolution from non-ECS	👍Possible	👍Possible	👍Possible	👿Not possible	👍Possible
🐙Cross-VPC Communication	👍Possible (requires VPC Peering)	👍👍Specialty	👿Not possible	👍Possible	👍Possible
🐙Cross-Account Communication	👍Possible (requires VPC Peering)	👍👍Specialty	👿Not possible	👿Not possible	👍Possible

Here are some example selection criteria:

"I want something with decent features, familiar, and won't disappear" -> ALB
"I use various services like EKS and Lambda besides ECS, and want to connect many VPCs/accounts" -> VPC Lattice
"Simple name resolution is enough. Reliability and observability are not needed or will be implemented separately" -> Service Discovery
"I want to connect ECS services with high reliability and observability. Communication with non-ECS services is minimal" -> Service Connect
"I have many microservices and strong requirements for reliability and observability. Need to connect non-ECS services too" -> App Mesh (Scheduled for deprecation, so consider VPC Lattice or Service Connect)

What is ECS Interservice Communication?

It refers to communication between ECS services.

It's not just about "how to find the access target" but also includes considerations like "what to do if the target is down" and "how to observe the communication".

In reality, there are cases where Lambda or EC2 needs to access ECS, so we'll touch on this as one of the comparison points.

Overview of Each Method

1. ALB

ALB is an L7 load balancer that can also connect to non-ECS services.
Its main strengths are being the most "mature" with abundant information and the confidence that "ALB won't be discontinued".

Connection Method

Create an ALB for service-to-service communication and access it using its domain name.

You can set multiple path-based routing rules on the ALB to avoid creating many ALBs.

Example: /web routes to service-web, /app routes to service-app

2. VPC Lattice

VPC Lattice is an application connectivity service that abstracts ALB functionality, which became GA in March 2023.
It can connect EC2, EKS, and Lambda, with cross-VPC and cross-account communication as its key strengths.

Connection Method

Access using the Lattice Service domain name + port,
which routes to ECS services associated with the Lattice Target Group defined in the Listener.

ECS Service Integration via VPC Lattice

Lattice Service Network: Logical collection of Lattice Services
Lattice Service: Similar to "one ALB", can have domain name and multiple Listeners
Lattice Target Group: Almost the same as ELB target groups, but not compatible

Before Native ECS Integration

Previously, ALB was required in front of ECS, but this became unnecessary with the November 18, 2024 update.

Previously: ALB was required in front of ECS

Associating Multiple ECS Services

Each Listener can have multiple ListenerRules like ALB.
By setting path-based routing, you can associate multiple ECS services with one Lattice Service.

Associating multiple ECS services with one Lattice Service

This improves manageability and potentially reduces costs.
As discussed later, there's a time-based charge per Lattice Service.

Other Features

IAM authentication can be used to restrict communication sources.
- https://docs.aws.amazon.com/vpc-lattice/latest/ug/auth-policies.html
TCP support allows access to DBs etc. (though expensive)
- https://aws.amazon.com/about-aws/whats-new/2024/12/vpc-lattice-tcp-vpc-resources/

3. ECS Service Discovery

ECS Service Discovery provides a simple name resolution mechanism.
While it lacks reliability and observability features, it offers simplicity and cost-effectiveness.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html

Connection Method

Client resolves <service namespace>.<hosted-zone name> to get backend Task IP addresses
Access using those IP addresses

Unlike other methods, containers can communicate directly with target services.

4. ECS Service Connect

Service Connect, launched in November 2022, combines features from ELB, Service Discovery, and App Mesh.
It specializes in ECS-to-ECS communication and provides easy setup for reliability and observability features.

Connection Method

An Envoy-based sidecar container (Service Connect Agent) handles communication.
It automatically handles Cloud Map registration, target discovery via Cloud Map, and access.

Side Note: Integration with Lambda and VPC Lattice

There are requests to integrate Service Connect with Lambda and VPC Lattice.
While this might be challenging due to architectural and conceptual reasons, if realized, it would significantly expand Service Connect's possibilities.

5. App Mesh (Scheduled for Deprecation)

App Mesh is a feature-rich service mesh service consisting of [1]managed Envoy control plane and [2]Envoy sidecar containers.

It's scheduled for deprecation by September 30, 2026. Two migration paths are recommended:

Connection Method

Like Service Connect, the sidecar handles various tasks.
Since it's scheduled for deprecation, we'll omit detailed mechanism explanations.

Comparison by Aspect

From here, we'll mostly exclude App Mesh.
It's scheduled for deprecation and is generally powerful except for its complexity.

Cost💰

	ALB	VPC Lattice	👑ECS Service Discovery	ECS Service Connect	App Mesh
Cost	Time + LCU	Time + traffic + request count	👍👍Very cost-effective	Sidecar	Sidecar

Service Discovery is extremely cost-effective as it doesn't require sidecars
ALB requires complex cost calculations based on LCU
VPC Lattice has time-based charges per Lattice Service
- Time-based charges alone cost approximately $23.4 per month per Service (\$0.0325/h x 24 x 30)
- For cost efficiency, it's better to associate multiple ECS services with one Lattice Service rather than creating a separate Lattice Service per ECS service (similar to ALB)
Service Connect charges for sidecar resources
- With minimum recommended specs for x86 Fargate, it costs about $9.1 per task per month (\$0.05056/h x 0.25 x 24 x 30)²
- Of course, larger sizes may be needed depending on traffic
- May be cost-prohibitive with many tasks

Observability🔭

	ALB	VPC Lattice	ECS Service Discovery	ECS Service Connect	App Mesh
Logs & Metrics	👍Supported	👍Supported	👿None	👍Supported	👍Supported
Tracing	👿None (ID is assigned)	👿None	👿None	👿None	👍Supported

Service Discovery is designed without observability features, and this is unlikely to change
Tracing is App Mesh's specialty, with Envoy sidecars capable of sending data to X-Ray or Datadog
Service Connect, being Envoy-based, might support tracing in the future
- There are feature requests for this https://github.com/aws/containers-roadmap/issues/2369
ALB adds X-Amzn-Trace-Id to request headers, but this alone doesn't enable rich distributed tracing

Reliability🛡️

	ALB	VPC Lattice	ECS Service Discovery	👑ECS Service Connect	App Mesh
Health Check	👍 Supported	👍Supported	👍Supported (container level)	👍Partially supported¹	👍Supported
Auto Retry	👿 None	👿None	👿None	👍Supported	👍Supported
Circuit Breaker	👿 None	👿None	👿None	Partial(outlier detection)	👍Supported

Health checks are essential, but auto retry and circuit breakers aren't mandatory
Service Connect has advantages in auto retry and outlier detection
- Auto retry retries on different tasks if the original task fails
App Mesh's circuit breaker consists of [1]outlier detection + [2]connection pool
- Outlier detection(outlier detection): Monitors actual traffic to stop routing to hosts with many errors
- Connection pool: Limits concurrent connections or requests

https://aws.amazon.com/about-aws/whats-new/2020/11/aws-app-mesh-introduces-circuit-breaker-capabilities/?nc1=h_ls

Deployment🚢

	👑ALB	VPC Lattice	ECS Service Discovery	ECS Service Connect	App Mesh
Supported Deployment Types	👍 All	Technically all	👍All	👿`ECS` only	👍All
Canary Deployment	👍 Easy	👍Easy	👿Tricky	👿Not possible	👍Easy

※ ECS deployment types are ECS(rolling update), CODE_DEPLOY(Blue/Green), and EXTERNAL(external deployment).

ALB is the most straightforward to handle. It's well integrated with ECS and has minimal constraints
VPC Lattice can perform Canary deployments similar to ALB by manipulating ListenerRules
- When using Internal-ALB, it's still ALB, so it supports all deployment types and enables flexible deployment
- However, with native ECS integration, only the ECS deployment type can be selected
Service Discovery supports all deployment types but lacks flexible routing control, making Canary deployments challenging
Service Connect can only use the ECS deployment type, limiting deployment flexibility

Connection Breadth🐙

	ALB	👑VPC Lattice	ECS Service Discovery	ECS Service Connect	App Mesh
Name Resolution from non-ECS	👍Possible	👍Possible	👍Possible	👿Not possible	👍Possible
Cross-VPC	👍Possible (requires VPC Peering)	👍👍Specialty	👿Not possible	👍Possible	👍Possible
Cross-Account	👍Possible (requires VPC Peering)	👍👍Specialty	👿Not possible	👿Not possible	👍Possible

This is VPC Lattice's core strength and unique selling point
ALB is a standard VPC resource, requiring VPC Peering or something for cross-VPC/account communication
Service Discovery cannot cross VPCs or accounts, but this helps maintain its simplicity
Service Connect is ECS-specific and has the most limitations
- A separate method is needed for the "initial entry point from outside to ECS"
- Cross-account communication might be supported in the future https://github.com/aws/containers-roadmap/issues/2148

Conclusion

Considering costs and complexity, "reducing ECS interservice communication in the first place" might also be an option.

I'll continue to watch the growth of VPC Lattice and Service Connect.

Currently, there are cases where "routing occurs before tasks become HEALTHY" https://github.com/aws/containers-roadmap/issues/2334 ↩
Minimum 256CPU(=0.25vCPU) and 64MiB additional resources per task are recommended for sidecars. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect-concepts-deploy.html#service-connect-concepts-proxy ↩

Introduction to Gitless GitOps: A New OCI-Centric and Secure Architecture

Tetsuya KIKUCHI — Thu, 17 Apr 2025 03:19:50 +0000

This article is translated and edited from my own article in Japanese.

Introduction

I learned about Gitless GitOps at KubeCon EU 2025. As a developer of a GitOps tool (PipeCD), I was intrigued but found limited information available, so I decided to investigate.

"GitOps without Git? What do you mean?"

You can read the details in this report. Gitless is introduced as part of D2, Flux's reference architecture, so you don't need to read the entire document.

https://fluxcd.control-plane.io/guides/d2-architecture-reference/

Key Points

Gitless GitOps is a new GitOps architecture that centers on OCI Registry instead of Git
While it's "Gitless" because GitOps tools don't access Git, the fundamental principles remain largely unchanged
The main benefit is enhanced security in the supply chain

Review: Current GitOps

What is GitOps?

GitOps is a delivery method that "keeps the production environment in sync with Git configuration."
It follows these 4 principles:

Declarative
Versioned and Immutable
Pulled Automatically
Continuously Reconciled

Popular GitOps tools include ArgoCD, FluxCD, and PipeCD.

Simple GitOps Architecture

Here's a simple GitOps architecture:

The GitOps tool (like FluxCD) continuously monitors the Git repository and deploys changes to the production environment when detected.

Overview of Gitless GitOps

Gitless GitOps is a delivery method driven by OCI (Open Container Initiative) artifacts stored in container registries instead of Git repositories.

CI creates and pushes OCI artifacts, while the GitOps tool focuses solely on applying them.
Continuous reconciliation is based on OCI artifacts rather than Git.

Note that Git remains the Single Source of Truth in Gitless, but mechanisms to maintain consistency between Git and OCI registries through write control are necessary.

This is unrelated to the tool called "Gitless": https://gitless.com/

Side Note: When Did It Emerge?

As far as I researched, the concept and term "Gitless" appears to have been led by FluxCD developers within the last 3 years.

2022: Flux added OCI support (OCIRepository introduced)
- https://fluxcd.io/blog/2022/10/cncf-talk-flux-oci/
2024: The term "Gitless GitOps" was already introduced
- https://www.heise.de/en/background/Gitless-GitOps-The-path-to-a-secure-app-environment-with-Flux-and-OCI-9811957.html
April 2025: ControlPlane released a reference architecture based on Gitless GitOps
- ControlPlane is a company providing enterprise solutions for FluxCD and other tools
- https://control-plane.io/posts/d2-reference-architecture-guide/

There seems to be variation in spelling between "Gitless" and "Git-less".

Benefits of Gitless

The underlying context for these benefits seems to be that "Git itself wasn't designed for GitOps, and OCI registries are more suitable for cloud-native environments."

Software Supply Chain Security and Compliance

This is the most emphasized benefit.
The KubeCon session included the message "Configuration is Part of the Supply Chain".

Gitless enables the following, with OCI ecosystem integration being a major advantage:

Manifest authenticity and integrity can be verified through OCI artifact signing/verification
- Flux uses cosign
Vulnerability scanning can be performed in OCI repositories
SBOMs can handle manifests as well

Background

The increased focus on such measures is driven by regulations including:

US: EO14028 "Executive Order on Improving the Nation's Cybersecurity" issued in 2021
Europe: Cyber Resilience Act (CRA) enacted in 2024

No Git Access Required

Eliminating git pull means network access, authentication, and permissions for Git are no longer needed by GitOps tools.
As mentioned later, write permissions to Git also become unnecessary.

Can authenticate to OCI registries using Workload Identity without PATs or SSH keys
- No need for secret management or rotation
Likely improves Git access permission management efficiency
- Prevents scenarios like "Strict Git access control inadvertently restricted GitOps tool permissions, breaking Image Updater functionality"

Performance

Gitless focuses on pulling OCI artifacts instead of git pull, potentially leading to lighter and faster operations.

Current GitOps can face performance challenges with large manifest repositories or frequent updates from trunk-based development. This is because GitOps tools frequently perform git pull and create local copies of pulled repositories.

Various GitOps tools implement performance tuning for git operations, but some aspects remain challenging to optimize, such as identifying which manifests are linked to applications due to templating. Here's an example from PipeCD:

https://pipecd.dev/blog/2024/09/11/performance-improvement-in-git-operations-on-pipecd-v0.48.9/

Simplification of GitOps Tools

While CI's role expands, the GitOps tool's responsibilities are reduced and simplified. GitOps tools focus solely on "deploying OCI artifacts."

[1] Templating Processing

Since CI executes kustomize, jsonnet, etc., and uses the results as OCI artifacts, templating processing becomes unnecessary in CD. The final manifests to be applied can be fixed regardless of GitOps tool versions or dependencies.

Flux's documentation mentions that OCI artifact-centric approaches are particularly convenient when final manifests aren't in Git due to templating.

[2] Automatic Image Updates

Various GitOps tools have mechanisms to automatically deploy new application versions. All involve GitOps tools committing to manifest repositories themselves:

FluxCD: Image Reflector Controller & Image Automation Controller
ArgoCD: Image Updater
PipeCD: EventWatcher

In Gitless, these responsibilities shift from GitOps tools to CI. This eliminates the need for Git write permissions for GitOps tools.

Here's the Before/After of the overall CI/CD pipeline including image updates:

Current architecture:

Gitless architecture:

Potentially Good for Edge Environments?

While not mentioned in ControlPlane's report, some suggest that replicating OCI repositories across locations could enable fast, stable deployments in edge environments.

https://itnext.io/advantages-of-storing-configuration-in-container-registries-rather-than-git-b4266dc0c79f

What's Needed to Migrate to Gitless?

Here are some necessary tasks for migrating to Gitless. Both GitOps tool users and developers need to make adjustments.

User-side Tasks

Reflect application image pushes in the manifest repository
Create, sign, and push OCI artifacts from the manifest repository
Modify GitOps tool configuration to monitor OCI artifacts instead of Git
- Git access permissions and network access for GitOps tools can be removed

Developer-side Tasks (Already Implemented in Flux)

Support OCI registry monitoring, retrieval, and signature verification
- For Flux, the main task was adding OCIRepository to SourceController, which appears to be cleanly implemented
Support OCI artifact deployment
- Since manifest apply functionality should already exist, mainly needs artifact extraction processing

ArgoCD is currently working on implementation, while PipeCD hasn't started yet.

For implementation, the OCI artifact manipulation client tool ORAS (CNCF Sandbox project) seems useful. ~~Flux uses oras-go.~~

[2025-04-17 edited] Flux implements their own OCI package here: https://github.com/fluxcd/pkg/tree/main/oci

Conclusion

FluxCD's pioneering presence is impressive.

Should everyone adopt Gitless? "Not everyone at this point" is my answer. While GitOps tools currently handle many tasks, Gitless requires additional CI setup.
However, Gitless could be powerful in industries and countries requiring strict security and compliance.

The momentum of Gitless likely depends on the strictness of software supply chain security regulations and the emergence of large-scale GitOps performance case studies.

While writing this article, unsettling news about CVE program continuity emerged...

ECS External Deployment & TaskSet - Complete Guide

Tetsuya KIKUCHI — Wed, 19 Mar 2025 11:35:00 +0000

This guide explores "External Deployment" and "TaskSet" - two of the lesser-known yet powerful features in ECS. We'll cover their concepts and practical usage, including real-world examples with PipeCD. Mastering these features enables advanced deployment strategies in ECS.

Note: This article is based on specifications as of December 14, 2024.

Key Takeaways

TaskSet: A layer between Service and Task that maintains revisions per TaskSet
External Deployment: A deployment type that enables flexible deployments using TaskSets
These features are essential for implementing advanced deployment strategies like Canary in ECS

Understanding the Terminology

1. What is External Deployment?

External Deployment is a deployment type in ECS that enables flexible deployments through third-party controllers.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-external.html

Note: This is unrelated to launchType: EXTERNAL used in ECS Anywhere.

Three Deployment Types

ECS offers three deployment types, with External Deployment being one of them:

ECS: Rolling update - the default and most conventional approach
CODE_DEPLOY: Integrates with CodeDeploy for straightforward Canary and Blue/Green deployments
EXTERNAL: Enables custom deployments through TaskSet manipulation

Important: The deployment type cannot be changed after service creation.

Differences between `CODE_DEPLOY` and `EXTERNAL`

Both manipulate TaskSets and follow similar operational flows[^1]
CODE_DEPLOY is easier to implement as CodeDeploy handles most operations
EXTERNAL offers more flexibility despite its complexity
- Examples: Custom deployment validation, rollback strategies

2. What is TaskSet?

TaskSet is a concept that sits between ECS Service and Task, grouping Tasks together.

TaskSets only appear when using CodeDeploy or External deployment types, handling Task scheduling and scaling instead of the Service.

You won't encounter TaskSets when using rolling updates or running Standalone Tasks.

Examining TaskSets

TaskSets aren't visible in the console. You can view them by using DescribeServices on a Service using CodeDeploy or External deployment. They have similar attributes to Services, including runningCount, launchType, and networkConfiguration.

$ aws ecs describe-services --cluster xxx --services yyy-service
{
    "services": [
        {
            "serviceName": "yyy-service",
            "taskSets": [ // Here
                {
                    "id": "ecs-svc/4390851359570905103",
                    "taskSetArn": "arn:aws:ecs:ap-northeast-1:<account>:task-set/xxx/yyy-service/ecs-svc/4390851359570905103",
                    "serviceArn": "arn:aws:ecs:ap-northeast-1:<account>:service/xxx/yyy-service",
                    "clusterArn": "arn:aws:ecs:ap-northeast-1:<account>:cluster/xxx",
                    "status": "ACTIVE",
                    "taskDefinition": "arn:aws:ecs:ap-northeast-1:<account>:task-definition/zzz-taskdef:1",
                    "computedDesiredCount": 2,
                    "pendingCount": 2,
                    "runningCount": 0,
                    "createdAt": "2024-12-13T20:00:24.064000+09:00",
                    "updatedAt": "2024-12-13T20:00:28.178000+09:00",
                    "launchType": "FARGATE",
                    "platformVersion": "1.4.0",
                    "platformFamily": "Linux",
                    "networkConfiguration": {
                        "awsvpcConfiguration": {
                            "subnets": [
                                "subnet-aaa",
                                "subnet-bbb"
                            ],
                            "securityGroups": [
                                "sg-ccc"
                            ],
                            "assignPublicIp": "DISABLED"
                        }
                    },
                    "loadBalancers": [],
                    "serviceRegistries": [],
                    "scale": {
                        "value": 100.0,
                        "unit": "PERCENT"
                    },
                    "stabilityStatus": "STABILIZING",
                    "stabilityStatusAt": "2024-12-13T20:00:24.064000+09:00",
                    "tags": []
                }
            ],
            "deploymentController": {
                "type": "EXTERNAL" // External deployment
            },
            ...
        }
    ],
}

When to Use TaskSets

TaskSets enable Canary releases and Blue/Green deployments because you can specify different TargetGroups and TaskDefinitions per TaskSet:

"TaskSet with v1 TaskDefinition points to TargetGroup A"
"TaskSet with v2 TaskDefinition points to TargetGroup B"

Control traffic distribution between TargetGroups at the ALB level.

Canary release and Blue/Green deployment using TaskSets

TaskSet Operations

There are only five APIs for TaskSet operations. Most operations involve managing multiple TaskSets rather than modifying a single TaskSet's internals.

CreateTaskSet
DescribeTaskSets
- Requires TaskSet ID beforehand, making it cumbersome
- DescribeServices is recommended for easier TaskSet information retrieval
DeleteTaskSet
UpdateTaskSet
- Can only update scale
  - scale: Percentage of Tasks to run in this TaskSet relative to Service's desiredCount
  - Example: With desiredCount 10 and scale 30%, TaskSet maintains 3 Tasks
  - Range: 0-100
  - Task count automatically adjusts when Service autoscales (changes in desiredCount)
- Cannot update taskDefinition, ensuring all Tasks within a TaskSet use the same image with immutable tags
UpdateServicePrimaryTaskSet
- Changes a TaskSet's status to PRIMARY
- Optional but recommended as Primary TaskSet cannot be deleted with DeleteTaskSet¹
- Making a TaskSet Primary changes the previous Primary TaskSet to ACTIVE
- Some Primary TaskSet settings propagate to Service (launchType, taskDefinition, etc.)

 - Before Primary TaskSet exists:

     ```json
     $ aws ecs describe-services --cluster xxx --services yyy-service
     {
         "services": [
             {
                 "serviceName": "yyy-service",
                 "launchType": "EC2", // Initial Service creation
                 "taskSets": [
                     {
                         "id": "ecs-svc/8721983045402263235",
                         "status": "ACTIVE", // Not PRIMARY yet
                         "taskDefinition": "arn:aws:ecs:ap-northeast-1:<account>:task-definition/zzz-taskdef:3",
                         "launchType": "FARGATE",
                         ...
     ```

 - After `UpdateServicePrimaryTaskSet`:

     ```json
     $ aws ecs describe-services --cluster xxx --services yyy-service
     {
         "services": [
             {
                 "serviceName": "yyy-service",
                 "launchType": "FARGATE", // Propagated from TaskSet
                 "taskDefinition": "arn:aws:ecs:ap-northeast-1:<account>:task-definition/zzz-taskdef:3", // Propagated from TaskSet
                 "taskSets": [
                     {
                         "id": "ecs-svc/8721983045402263235",
                         "status": "PRIMARY", // Now PRIMARY
                         "taskDefinition": "arn:aws:ecs:ap-northeast-1:<account>:task-definition/zzz-taskdef:3",
                         "launchType": "FARGATE",
                         ...
     ```

Example: Canary Release with External Deployment

There are multiple approaches to External Deployment, including rollback strategies.
Here are two examples.
For more details, refer to this documentation.

Note: This article focuses on Services behind ALB.

Method A: Promoting Canary to Primary

Prerequisites

Assume ALB, ListenerRule, two TargetGroups, ECS Cluster, and TaskDefinitions (v1,v2) are ready

Step 1: Create Service

Use CreateService to create a Service.

[Required] Set deploymentController.type to EXTERNAL
Fewer required parameters compared to ECS deployment type
- No need for taskDefinition, launchType, networkConfiguration
- Details here

$ aws ecs create-service \
--cluster <cluster-name> \
--service-name <service-name> \
--desired-count 2 \
--deployment-controller type=EXTERNAL

No TaskSets or Tasks are created at this point.

Step 2: Create Initial TaskSet (v1)

Use CreateTaskSet to create the first TaskSet.

Specify parameters typically set at Service level (launchType, taskDefinition, etc.)
loadBalancers[].targetGroupArn: Specify TargetGroup for ALB traffic
Recommend 100% scale for initial TaskSet

$ aws ecs create-task-set \
--cluster <cluster-name> \
--service <service-name> \
--task-definition <task-def-arn-v1> \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[subnet-aaa,subnet-bbb],securityGroups=[sg-ccc],assignPublicIp=ENABLED}" \
--load-balancers "targetGroupArn=<tg-arn-1>,containerName=web,containerPort=80" \ # Match TaskDefinition
--scale unit=PERCENT,value=100

After prerequisites

Note: UpdateServicePrimaryTask is optional here

Deployment Flow

1. Create TaskSet (v2)

Similar to Step 2 above, but with different task-definition, targetGroupArn, and scale.

$ aws ecs create-task-set \
--cluster <cluster-name> \
--service <service-name> \
--task-definition <task-def-arn-v2> \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[subnet-aaa,subnet-bbb],securityGroups=[sg-ccc],assignPublicIp=ENABLED}" \
--load-balancers "targetGroupArn=<tg-arn-2>,containerName=web,containerPort=80" \ # Match TaskDefinition
--scale unit=PERCENT,value=10

No traffic flows to v2 TaskSet yet as its TargetGroup weight is 0.

Creating Canary TaskSet

2. Route Traffic to Canary

Use ELB's ModifyRule to adjust TargetGroup weights.

$ aws elbv2 modify-rule \
--rule-arn <listener-rule-arn> \
--actions '[{
    "Type": "forward", 
    "ForwardConfig": {
        "TargetGroups": [
            {
                "TargetGroupArn": "<tg-arn-1>", 
                "Weight": 90
            },
            {
                "TargetGroupArn": "<tg-arn-2>", 
                "Weight": 10
            }
        ]
    }
}]'

Starting traffic flow to Canary

Use ModifyListener for default listener rules.
For multiple ListenerRules, gather all target rules beforehand.

3. If Successful, Route 100% Traffic to Canary

Note: If Canary's scale was less than 100, first use UpdateTaskSet to set it to 100.

Use ELB's ModifyRule again to update TargetGroup weights.

$ aws elbv2 modify-rule \
--rule-arn <listener-rule-arn> \
--actions '[{
    "Type": "forward", 
    "ForwardConfig": {
        "TargetGroups": [
            {
                "TargetGroupArn": "<tg-arn-1>", 
                "Weight": 0
            },
            {
                "TargetGroupArn": "<tg-arn-2>", 
                "Weight": 100
            }
        ]
    }
}]'

Routing 100% traffic to Canary

4. Promote Canary TaskSet to Primary

4.1. Get Canary TaskSet ID

$ aws ecs describe-services \
--cluster <cluster-name> \
--services <service-name> \
| jq '.services[0].taskSets[]'

Find Canary TaskSet ID in response, using status (PRIMARY/ACTIVE) to identify it.

4.2. Use `UpdateServicePrimaryTaskSet` to Make Canary TaskSet Primary

$ aws ecs update-service-primary-task-set \
--cluster <cluster-name> \
--service <service-name> \
--primary-task-set <ecs-svc/xxx> # Canary TaskSet ID from 4.1

5. Delete Old TaskSet

Use DeleteTaskSet to remove v1 TaskSet.

$ aws ecs delete-task-set \
--cluster <cluster-name> \
--service <service-name> \
--task-set <ecs-svc/xxx> # Old TaskSet ID from 4.1

After deployment completion

Rollback Scenarios

If failure occurs during steps 2 or 3:
1. Return 100% traffic to TargetGroup A
2. Delete Canary TaskSet
If failure occurs during step 5:
1. Create new TaskSet using TaskDefinition v1 and TargetGroup A
2. Update it to Primary
3. Route 100% traffic to TargetGroup A
4. Delete Canary TaskSet

Method A Challenges

TargetGroup assignments to old/new TaskSets aren't fixed, requiring tracking of current assignments.
This complicates rollback especially.

Method B: Using Three TaskSets

Here's an alternative approach that addresses Method A's challenges.
It uses three TaskSets: "current", "canary", and "new".

This is the approach PipeCD adopts.

Deployment Flow

0. Initial State

1. Create Canary

Use CreateTaskSet and elbv2::ModifyRule.
Canary TaskSet's scale can be less than 100%.

2. If Successful, Create New TaskSet

Use CreateTaskSet with TargetGroup A and TaskDefinition v2.

Traffic now flows to the third TaskSet.

3. Delete All Old TaskSets

Use UpdateServicePrimaryTaskSet, elbv2::ModifyRule, and DeleteTaskSet.

Back to initial state configuration.
Next deployment starts with TargetGroup A pointing to latest version.

Rollback Scenarios

For any failure phase, return to initial state:

Create new TaskSet with TaskDefinition v1 and TargetGroup A
Update it to Primary
Route 100% traffic to TargetGroup A
Delete all other TaskSets

For failures before step 3, faster rollback:

Route 100% traffic to TargetGroup A
Delete all other TaskSets

Pros/Cons

Pros:

Fixed TargetGroup assignments simplify operations

Cons:

Longer deployment time due to two TaskSet creations
Higher cost with three TaskSets

Works on EC2 Too

Method B works on EC2 as well as Fargate.

Initially concerned about deploymentConfiguration.maximumPercent limitations
Worried about "Can't use Method B if limited to 2x desiredCount Tasks", but maximumPercent is ignored
Note: Docs explicitly state that maximumPercent is ignored for External Deployment on Fargate

Using Method B with PipeCD

External Deployment can be complex (especially with rollbacks).
PipeCD implements Method B Canary releases with simple pipeline definitions.
Handles rollbacks automatically.

  pipeline:
    stages:
      # 1. Create Canary TaskSet
      - name: ECS_CANARY_ROLLOUT 
        with:
          scale: 10
      # 2. Route 10% traffic to Canary
      - name: ECS_TRAFFIC_ROUTING 
        with:
          canary: 10
      # 3. Approval phase
      - name: WAIT_APPROVAL
      # 4. Create new Primary TaskSet
      - name: ECS_PRIMARY_ROLLOUT 
      # 5. Route 100% traffic to new Primary TaskSet
      - name: ECS_TRAFFIC_ROUTING 
        with:
          primary: 100
       # 6. Delete unnecessary TaskSets
      - name: ECS_CANARY_CLEAN

Full configuration file here.

Important Considerations for External Deployment

Several important points to consider when using External Deployment.

Many Service Features/Settings Unsupported

Examples of unsupported features:
- Service Connect
- VPC Lattice (possible via Internal-ALB)
- Deployment Circuit Breaker
- Deployment Alarms
No comprehensive list of unsupported features exists
- Recommend searching docs for keywords like "Deployment"
- Service Connect example: > Only services that use rolling deployments are supported with Service Connect.
No known TaskDefinition-specific constraints for External Deployment

External Deployment Often Excluded from ECS Updates

November 2024's "Improved Deployment Visibility" enhanced ECS deployment history UI/API, but External Deployment remained unchanged
November 2024's VPC Lattice-ECS native integration still requires Internal-ALB for External Deployment

Migration from Other Deployment Types is Challenging

As mentioned, deployment type can't be changed after Service creation (particularly painful)
- Migrating to External Deployment requires creating new Service and gradually shifting traffic
Many unsupported features mean reviewing Service settings
- Some settings move from CreateService to CreateTaskSet

Manual TaskSet Tagging Required

TaskSet doesn't support tag propagation (propagateTags) from Service/TaskDefinition reference
enableECSManagedTags doesn't apply Managed tags to TaskSets

Limited Console Information

TaskSets not visible in ECS console, requiring CLI usage
"Deployment History" remains unchanged for External Deployment
Reference: PipeCD visualizes Service/TaskSet/Task states in UI ²

PipeCD UI (during Canary deployment)

Other Constraints

Can't link multiple TargetGroups to one TaskSet

Maximum 5 TaskSets per Service

Allows up to 5x desiredCount Tasks per Service (scale100 x 5 TaskSets)
Not documented
Attempting to exceed limit results in:

An error occurred (InvalidParameterException) when calling the CreateTaskSet operation: 5 deployments exist on the service. Unable to create new TaskSet since this exceeds the maximum limit.

Additional Notes

Note 1: Why Low Awareness?

Two hypotheses for External Deployment and TaskSet's low recognition:

Minimal console presence

Service creation screen - "External Deployment" not visible
Satisfaction with ECS/CODE_DEPLOY deployment types (but are they truly sufficient?)

Note 2: Do TaskSets Exist in `ECS` Deployment Type?

A: Possibly used internally, but unclear

Two hints suggest TaskSet-like behavior:

Task's StartedBy format matches External Deployment's ecs-svc/xxx
- TaskSet IDs use ecs-svc/xxx format, and Tasks within TaskSets show this as StartedBy
The xxx in ecs-svc/xxx matches service revision ID

Service revision mechanism seems similar to TaskSets but has different ARN format.

DescribeTaskSets on ecs-svc/xxx fails:

$ aws ecs describe-task-sets --cluster aaa --service bbb --task-sets ecs-svc/xxx

An error occurred (InvalidParameterException) when calling the DescribeTaskSets operation: Amazon ECS only supports task set management on services configured to use external deployment controllers.

Conclusion

External Deployment, while sparsely documented and constrained, is powerful when mastered.
Service Connect and VPC Lattice support would be welcome additions.

While migrating to External Deployment is challenging, PipeCD will soon support other deployment types through "plugins".
For plugin details, see:

https://pipecd.dev/blog/2024/11/28/overview-of-the-plan-for-pluginnable-pipecd/

DeleteTaskSet will return this error: An error occurred (InvalidParameterException) when calling the DeleteTaskSet operation: Primary TaskSet cannot be deleted ↩
https://pipecd.dev/docs/user-guide/managing-application/application-live-state/ ↩

ecstop: My CLI Tool to Stop ECS Resources Easily

Tetsuya KIKUCHI — Sun, 05 Jan 2025 15:42:27 +0000

Introduction

On December 20, 2024, I developed and released a CLI tool called ecstop that "quickly stops ECS resources in bulk".

https://github.com/t-kikuc/ecstop

The name ecstop is a combination of ECS+Stop. I pronounce it as "ee-c-stop".

In this article, I'll introduce the overview, philosophy, and future prospects of ecstop.

Summary in 3 Lines

ecstop can quickly stop ECS services, tasks, and container instances (EC2) in bulk.
The main purpose is to easily reduce costs in dev environments.
After brew install t-kikuc/tap/ecstop, you're ready to use immediately without any configuration file.

Development Background

I often created ECS resources when testing ECS itself or developing/testing PipeCD.

Since ECS charges for running tasks and container instances, I wanted to stop unused ones.

As it's for testing, I didn't want to delete clusters or services. They're free.

However, as I created many resources, it was troublesome to stop them one by one from the AWS console.

To stop a service from the console, you need to go to the ECS console, select the service and "update", set the number of tasks to 0 and "confirm update" for each service.
IaC and deployment tools are designed for production use and require careful configuration file changes, which isn't suitable for "quickly stopping multiple resources".

I had been using shell scripts/Go to stop them in bulk, but I often forgot how to call them.
Therefore, I decided to make a proper CLI tool.

Installation

You can install ecstop with the following command (Homebrew):

$ brew install t-kikuc/tap/ecstop

To enable auto-completion, please refer to here.

Features

For details on options, please refer to the README.

1. Zero-scale Services

$ ecstop services --cluster xxx

This sets the desiredCount of all services in the xxx cluster to 0. This also automatically stops tasks linked to the services.

2. Stop Tasks

$ ecstop tasks --cluster xxx --standalone

The --standalone flag stops tasks that are not linked to services.

This applies to tasks whose group prefix is not service:.

To stop tasks linked to services, use ecstop services instead because services can start new tasks even after stopping by ecstop tasks.

3. Stop Container Instances (EC2)

$ ecstop instances --cluster xxx

This stops (≠ Terminate) all container instances linked to the xxx cluster.

4. Execute the Above Three in One Command

$ ecstop all --cluster xxx

This command is equivalent to:

$ ecstop services --cluster xxx
$ ecstop tasks --cluster xxx --standalone
$ ecstop instances --cluster xxx

Other Useful Flags

--all-cluster: Instead of --cluster xxx, this performs stop operations on all ECS clusters in the region.
--profile yyy: You can specify the AWS profile.
--region zzz: You can specify the AWS region.

Philosophy

1. No Deletion

ecstop does not delete unnecessary ECR images or task definitions.

While tasks and container instances (EC2) have high operating costs, ECR images are relatively inexpensive, and task definitions are free, so I ignored them. Services and clusters themselves are also free, so they are not deleted.

For this reason, I didn't include "clean" or "delete" in the tool name.

In cases where "I want to delete it because it's not being used and is an eyesore", it's easier to select multiple items from the AWS console and delete them after human judgment.

Also, for cleaning up ECR images, it's good to use a tool called ecrm created by fujiwara-san.

https://github.com/fujiwara/ecrm

2. Specialized for Bulk Operations

I've minimized the selector options for "which resources to stop".

The AWS console is sufficient for stopping individual resources, and in a testing environment (not staging, etc.), there shouldn't be any resources that "absolutely must not be stopped". Especially at night, let's stop them.

Future Prospects

I want to regularly execute ecstop as a scheduled process on AWS (e.g., daily at 24:00).
- It's troublesome to call it every time + AWS authentication is cumbersome from local.
- I'm thinking of creating IaC for EventBridge+Lambda.
- As it's for testing environments, completion notifications seem unnecessary. I'm not sure about error notifications.
At the moment, I don't have any "I want to add this option".
- If there are any requests, I'll consider them.

Main Tools I Used

ecstop is based on a typical Go CLI development stack.

Conclusion

This was my first time publishing a CLI tool, and I learned a lot, including how to define the philosophy.

If you have any feedback or requests, please let me know at https://github.com/t-kikuc/ecstop.

Actually, I'm developing another CLI tool related to ECS external deployment, which I plan to release soon.

Copy changed files to another branch in Git

Tetsuya KIKUCHI — Fri, 27 Dec 2024 17:40:00 +0000

git checkout <branch> -- path/to/file

By the above command, you can pickup files to copy to another branch.

For example,

You've been coding on branch-A for a long time.
You wanna open a PR from some files of them.
Then, execute:

# Switch back to main
git checkout main 
# Create and switch
git checkout -b branch-B 
# Import subset files to branch-B
git checkout branch-A -- path/to/file

Hello, test post

Tetsuya KIKUCHI — Fri, 27 Dec 2024 17:16:37 +0000

This is my first post.

How to write a blog post...

DEV Community: Tetsuya KIKUCHI

ECS Native Blue/Green is Here! With Strong Hooks and Dark Canary

Key Points

Update Overview

Previous Situation

Benefits of This Update

Detailed Features

Deployment Lifecycle Hooks

Lifecycle Stages

Event Payload

Function Return Values

Note: Partially Available in Rolling Update??

Test Listener / Listener Rule (Dark Canary)

Benefits

Usage

Additionally, Deployment Controller Now Updatable Post-Creation

Signs of CODE_DEPLOY Type Deprecation Not CodeDeploy itself

Benefits of This Update

Note

How It Works (ALB Case)

Deployment Flow

During Rollback

Hands-On Testing

1. Service Update

2. Deployment

2-1. Test Traffic

2-2. Production Traffic Switch

2-3. Bake Time

Deployment Status Monitoring

3. Testing Hook Failure

Notes

Note: "Rolling Update" Deployment Type Naming Issue

Conclusion

Comparing 5 Methods of ECS Interservice Communication Including VPC Lattice

Introduction

Background

Summary

What is ECS Interservice Communication?

Overview of Each Method

1. ALB

Connection Method

2. VPC Lattice

Connection Method

Before Native ECS Integration

Associating Multiple ECS Services

Other Features

3. ECS Service Discovery

Connection Method

4. ECS Service Connect

Connection Method

Side Note: Integration with Lambda and VPC Lattice

5. App Mesh (Scheduled for Deprecation)

Connection Method

Comparison by Aspect

Cost💰

Observability🔭

Reliability🛡️

Deployment🚢

Connection Breadth🐙

Conclusion

Introduction to Gitless GitOps: A New OCI-Centric and Secure Architecture

Introduction

Key Points

Review: Current GitOps

What is GitOps?

Simple GitOps Architecture

Overview of Gitless GitOps

Side Note: When Did It Emerge?

Benefits of Gitless

Software Supply Chain Security and Compliance

No Git Access Required

Performance

Simplification of GitOps Tools

Potentially Good for Edge Environments?

What's Needed to Migrate to Gitless?

User-side Tasks

Developer-side Tasks (Already Implemented in Flux)

Conclusion

ECS External Deployment & TaskSet - Complete Guide

Key Takeaways

Signs of `CODE_DEPLOY` Type Deprecation Not CodeDeploy itself

Differences between `CODE_DEPLOY` and `EXTERNAL`

4.2. Use `UpdateServicePrimaryTaskSet` to Make Canary TaskSet Primary

Note 2: Do TaskSets Exist in `ECS` Deployment Type?