DEV Community

Manu Shukla
Manu Shukla

Posted on Originally published at ecorpit.com

ECS AGENT_CONNECTIVITY auto-repair 2026: one enum, two AWS docs, no published threshold

ECS AGENT_CONNECTIVITY auto-repair 2026: one enum, two AWS docs, no published threshold

Summary. Amazon ECS now monitors whether the container agent is still in contact with the ECS control plane and marks the instance impaired when it is not. The health check type is called AGENT_CONNECTIVITY, and as of 25 August 2026 it is documented inconsistently across four AWS pages. The AWS CLI 2.36.30 command reference lists four possible values for the container-instance health check type field, and the ECS API Reference page for InstanceHealthCheckResult lists three. AGENT_CONNECTIVITY is the one that is missing from the second list. The developer guide page that documents container instance health also lists three types, still cites agent version 1.57.0 and AWS CLI 1.22.3 or 2.3.6 as the floor, and still shows a sample payload timestamped 2021-11-10T03:30:26+00:00. The one number that decides whether your instance gets drained, the disconnection threshold, appears on none of those pages.

That gap matters more than it sounds. AWS's own 2023 Containers blog states that the ECS agent "disconnects and reconnects several times per hour" as normal behaviour, and the reference solution in that post routes every disconnect event through an Amazon SQS delay queue precisely to avoid acting on transient drops. The new capability acts on the same signal without telling you how long it waits.

What changed, and where it is written down

Four pages in the ECS documentation set now describe overlapping pieces of the same feature, and they do not say the same thing.

The container instance health change events page is the only page that describes the new behaviour in prose. It says Amazon ECS "continuously monitors the connectivity between the container agent and the Amazon ECS control plane" and that "when a container instance remains disconnected beyond a threshold, Amazon ECS marks the instance as impaired." The page carries a sample EventBridge payload with "type": "AGENT_CONNECTIVITY", "status": "IMPAIRED", a statusReason of Agent disconnected since 2026-07-29T22:34:13Z, and an overallStatus of IMPAIRED.

The container instance health monitoring page enumerates the health check types you can expect back from describe-container-instances. It lists CONTAINER_RUNTIME, ACCELERATED_COMPUTE and DAEMON. It does not list AGENT_CONNECTIVITY. Its remediation advice, unchanged, is to wait and re-run describe-container-instances, check the EC2 console, review CloudWatch metrics and check the AWS Health Dashboard. Draining and replacement are not mentioned.

Health check type AWS CLI 2.36.30 reference ECS API Reference and developer guide
CONTAINER_RUNTIME Listed Listed
ACCELERATED_COMPUTE Listed Listed
DAEMON Listed Listed
AGENT_CONNECTIVITY Listed Absent
overallStatus values OK, IMPAIRED, INSUFFICIENT_DATA, INITIALIZING OK, IMPAIRED, INSUFFICIENT_DATA, INITIALIZING

The CLI reference is published per CLI version, 2.36.30 at the time of writing, and already carries the fourth value. The API Reference page and the developer guide page do not. When two AWS references for the same field disagree, the safe assumption is that the service can return the wider set.

The consequence for typed SDKs

The API Reference documents InstanceHealthCheckResult.type as a string with an enumerated set of valid values, and strongly typed SDKs generate constants from that set. If your Java, Kotlin, Go or .NET client was generated from a model that predates the fourth value, a returned AGENT_CONNECTIVITY will not match any generated constant. Any exhaustive switch over health check types falls through to its default branch. Code that treats an unrecognised type as "no problem" stops paging without telling you.

This is the part worth checking before the next on-call rotation, not after. Regenerate or upgrade the SDK, then grep for every place your code compares a health check type against a literal string or an enum constant.

Fargate is listed for the check and excluded from the summary event

The events page gives an availability line under each scenario. Read them together and a gap opens.

Event scenario Launch types listed Fargate covered
Container runtime health change EC2 No
Accelerated compute device health change ECS Managed Instances No
Daemon health change ECS Managed Instances No
Agent connectivity loss ECS Managed Instances, AWS Fargate, Amazon EC2 Yes
Overall health status change ECS Managed Instances and EC2 No

So on Fargate you can receive an AGENT_CONNECTIVITY health change event, and you cannot receive the overall-status-change event that AWS describes as the aggregate view. If your EventBridge rule matches on the overall status transition rather than on the individual health check, your Fargate tasks are outside the rule.

There is a second problem underneath that one. Everything else about this feature is expressed through container instances. DescribeContainerInstances accepts only TAGS and CONTAINER_INSTANCE_HEALTH in its include parameter, and its response model is built around ec2InstanceId, agentConnected and versionInfo.agentVersion. The AWS CLI reference describes the object it returns as "an Amazon EC2 or External instance that's running the Amazon ECS agent and has been registered with a cluster." Fargate tasks do not register container instances. There is no documented API call that shows you Fargate agent-connectivity health at all, only an event you have to catch in flight.

What actually breaks: draining that cannot finish

The auto-repair path is drain, replace, terminate. AWS documents that shape on the daemon auto repair page: ECS marks the instance draining, provisions a replacement, waits for health, moves the application tasks, then terminates the original. That is a good design. It also inherits every constraint on the container instance draining page, and two of them bite hard.

First, the deadlock. If a service has minimumHealthyPercent and maximumPercent both set to 100 percent, AWS states plainly that "the service can't remove existing tasks, and also cannot start replacement tasks. This prevents successful container instance draining and prevents making new deployments." A drain triggered by a transient agent disconnect on such a service does not complete. It sits.

Second, the clock. For ECS Managed Instances, draining runs a two-phase termination. Phase one is graceful. Phase two enforces a hard deadline "typically set to draining initiation time plus seven days," after which ECS force-deregisters and kills every remaining task regardless of completion status. Seven days of a stuck drain is seven days of an instance that accepts no new tasks and never leaves.

Third, and only for teams on Spot: the same page warns that if you use Spot Instances with minimumHealthyPercent at or above 100 percent, "the service will not have enough time to replace the task before the Spot Instance terminates."

The real risk here is not the feature. It is a correct feature firing on a signal AWS itself calls routine, into a deployment configuration that cannot drain.

Who is affected, and how to tell in ten minutes

Run three checks.

Check the deployment configuration on every ECS service, because that is where the deadlock lives:

aws ecs list-services --cluster your-cluster --query 'serviceArns' --output text \
  | xargs -n10 aws ecs describe-services --cluster your-cluster --services \
  | grep -A3 deploymentConfiguration
Enter fullscreen mode Exit fullscreen mode

Any service with minimumHealthyPercent of 100 and maximumPercent of 100 is a candidate for a drain that never finishes.

Check the agent version floor. The developer guide still names 1.57.0 as the minimum for automated container instance checks, and the agent update page notes that agent updates do not apply to Windows container instances at all, so Windows clusters need new instances rather than an in-place update. The ECS EC2 container instances page confirms that Docker support is currently limited to the last two major versions published on Amazon Linux, 20.10.x and 25.x.

Check your EventBridge rules. If they filter on detail.overallStatus, add a branch that matches detail.healthChecks[].type equal to AGENT_CONNECTIVITY, because the overall-status event is not emitted for Fargate.

What to do this week

Set the deployment configuration first. Moving maximumPercent to 200 on services that can tolerate double capacity briefly removes the deadlock condition entirely, and it is a one-field change.

Keep a manual escape hatch. The Managed Instances troubleshooting guide still tells operators to restart the agent with nsenter -t 1 -m -p systemctl restart ecs or to force-deregister the instance with deregister-container-instance --force. Neither instruction mentions automatic repair. Both still work, and you will want them the first time an auto-drain stalls.

Do not build alerting on raw disconnect events. AWS solutions authors Henrique Santana and Pablo Di Sabato wrote in the AWS Containers blog that "as a part of its normal operation the Amazon ECS Agent disconnects and reconnects several times per hour," and their reference architecture inserts an SQS delay queue for exactly that reason. If you page on every AGENT_CONNECTIVITY transition to IMPAIRED, you will page on noise.

Cost matters here too, because auto-repair replaces instances and replacement instances carry the ECS management fee. AWS cut G-series ECS Managed Instances management fees by 35 percent and P-series and AWS Trainium fees by 60 percent effective 1 July 2026, applied automatically with no customer action. If you are sizing the blast radius of a replacement loop on accelerated capacity, use the current rates, not last quarter's. The same reasoning applies to the ECS Managed Instances GPU fee cut and to how you model automated node replacement on EKS, where the drain-and-replace pattern is older and better documented.

India-specific considerations

Cluster headroom decides whether any of this hurts. A service pinned at minimumHealthyPercent 100 on a three-instance cluster has nowhere to place replacement tasks, and the smaller the cluster the sooner that bites. Teams running ECS out of ap-south-1 or ap-south-2 on a single small capacity provider should confirm spare capacity before enabling anything that drains automatically.

Teams handling personal data under the Digital Personal Data Protection Act 2023 should also note that automatic instance replacement destroys local instance state, including agent logs at /var/log/ecs/ecs-agent.log. If your incident evidence lives on the instance, ship it off the box before auto-repair takes the box away. Centralising CloudWatch log groups is the straightforward answer.

What is still unknown

Three things, all of which AWS could settle with one sentence each.

The threshold. The events page says "beyond a threshold" and gives no duration. Until AWS publishes it, you cannot calculate how long a network partition has to last before your instance is condemned.

Whether the auto-repair applies to EC2 launch type as well as Managed Instances. The daemon auto-repair page sits inside the Managed Instances section of the guide. The AGENT_CONNECTIVITY availability line names all three launch types. Those are different scopes.

Whether AGENT_CONNECTIVITY will be added to the API Reference enum or removed from the CLI model. Right now the two disagree, and the CLI is the one your code will meet in production.

FAQ

What is AGENT_CONNECTIVITY in Amazon ECS?

It is a container instance health check type that reports whether the ECS container agent is still in contact with the ECS control plane. When an instance stays disconnected past an unpublished threshold, Amazon ECS marks it impaired and emits an ECS Container Instance Health Change event on Amazon EventBridge.

Why do two AWS pages list different health check types?

The AWS CLI 2.36.30 command reference is generated from the live ECS service model and lists four values including AGENT_CONNECTIVITY. The ECS API Reference page for InstanceHealthCheckResult and the container instance health developer guide page are hand maintained and still list only three. Trust the CLI reference.

Does this work on AWS Fargate?

Partly. The events page lists ECS Managed Instances, AWS Fargate and Amazon EC2 for the agent connectivity scenario. It lists only ECS Managed Instances and EC2 for the overall health status change event. There is also no documented API call that returns Fargate agent connectivity health outside of the event itself.

What deployment setting can stop a drain from completing?

A service with both minimumHealthyPercent and maximumPercent set to 100 percent. AWS states that such a service cannot remove existing tasks and cannot start replacement tasks, which prevents container instance draining and blocks new deployments. Raising maximumPercent to 200 removes the deadlock condition.

How long does a stuck drain last on ECS Managed Instances?

The draining page describes a two-phase termination. Phase one is graceful completion. Phase two enforces a hard deadline typically set to draining initiation time plus seven days, after which ECS force-deregisters the instance and terminates all remaining tasks regardless of their completion status.

What agent version do I need?

The developer guide names container agent version 1.57.0 or later for automated container instance health checks, along with AWS CLI version 1.22.3 or later, or AWS CLI version 2.3.6 or later. Agent updates do not apply to Windows container instances, which need replacement rather than an in-place update.

Is a disconnected ECS agent always a problem?

No. AWS states in its Containers blog that the ECS agent disconnects and reconnects several times per hour during normal operation. That is why the AWS reference monitoring solution routes disconnect events through an Amazon SQS delay queue before alerting, so that transient reconnections never reach an operator.

What does instance replacement cost on accelerated capacity?

Replacement instances carry the ECS Managed Instances management fee. Effective 1 July 2026, AWS reduced G-series ECS management fees by 35 percent and P-series and AWS Trainium fees by 60 percent, applied automatically with no customer action required for workloads already running GPU instances on ECS Managed Instances.

How eCorpIT can help

eCorpIT runs container platform reviews for teams on Amazon ECS and Amazon EKS, covering deployment configuration, capacity headroom, EventBridge rule coverage and the SDK-level enum handling described above. Our AIOps and SRE incident automation practice builds the delay-and-confirm logic that stops transient agent disconnects becoming pages, and we compare build against buy for on-call tooling in our note on the AWS DevOps Agent on-call economics. eCorpIT is CMMI Level 5, MSME Certified and ISO 27001:2022 certified. To have your ECS deployment configuration audited against the drain deadlock, book a container platform review.

References

  1. Monitor Amazon ECS container instance health, Amazon ECS Developer Guide, retrieved 25 August 2026.
  2. Amazon ECS container instance health change events, Amazon ECS Developer Guide, retrieved 25 August 2026.
  3. InstanceHealthCheckResult, Amazon ECS API Reference, retrieved 25 August 2026.
  4. describe-container-instances, AWS CLI 2.36.30 Command Reference, retrieved 25 August 2026.
  5. DescribeContainerInstances, Amazon ECS API Reference, retrieved 25 August 2026.
  6. Daemon auto repair, Amazon ECS Developer Guide, retrieved 25 August 2026.
  7. Draining Amazon ECS container instances, Amazon ECS Developer Guide, retrieved 25 August 2026.
  8. Troubleshooting Amazon ECS Managed Instances, Amazon ECS Developer Guide, retrieved 25 August 2026.
  9. Amazon ECS EC2 Container Instances, Amazon ECS Developer Guide, retrieved 25 August 2026.
  10. Updating the Amazon ECS container agent, Amazon ECS Developer Guide, retrieved 25 August 2026.
  11. Monitoring the Amazon ECS Agent, Henrique Santana and Pablo Di Sabato, AWS Containers blog, 13 January 2023, updated 15 May 2024.
  12. Amazon ECS Managed Instances reduces GPU management fees by up to 60%, AWS What's New, 7 July 2026.

Last updated: 25 August 2026.

Top comments (0)