AWS CodeDeploy: How to Fix the "Cannot Reach Instance Service" Error

If you're a DevOps engineer, System engineer, or Cloud engineer using AWS CodePipeline, CodeBuild, and CodeDeploy to deploy applications to EC2 instances, you may encounter a deployment failure after a successful build. If the logs from CodeDeploy show the following error:

CodeDeploy agent was not able to receive the lifecycle event. Check the CodeDeploy agent logs on your host and make sure the agent is running and can connect to the CodeDeploy server.

Don’t panic! This guide will help you troubleshoot and resolve the issue.

Step 1: Check the Status of the CodeDeploy Agent on the EC2 Instance

The first step is to verify if the CodeDeploy agent is running on your EC2 instance. To check the status, run the following command:

sudo service codedeploy-agent status

If the agent is stopped, start it by running:

sudo service codedeploy-agent start

Step 2: Confirm IAM Role Permissions

Ensure that the EC2 instance has an IAM role attached to it, and this role must have the necessary permissions to interact with AWS CodeDeploy. The policy should include actions for CodeDeploy, S3, and CloudWatch Logs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "codedeploy:*",
        "s3:GetObject",
        "s3:ListBucket",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Step 3: Validate IAM Role Trust Relationship

Next, ensure that the IAM role trust relationship is set up correctly. It should allow EC2 instances to assume the role. The trust relationship policy should look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Step 4: Restart the CodeDeploy Agent

Once you've confirmed that the IAM role is correctly configured, restart the CodeDeploy agent:

sudo service codedeploy-agent restart

Step 5: Check the CodeDeploy Agent Logs

If the problem persists, inspect the CodeDeploy agent logs for additional error messages that might provide insight into why the lifecycle event failed. To tail the log file:

sudo tail -f /var/log/aws/codedeploy-agent/codedeploy-agent.log

Look for any errors similar to this one:

ERROR [codedeploy-agent(3313518)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Cannot reach InstanceService: Aws::CodeDeployCommand::Errors::AccessDeniedException - Aws::CodeDeployCommand::Errors::AccessDeniedException

Step 6: Remove AWS Credentials from the Instance (if applicable)

If the error mentions AccessDeniedException, it's possible that an AWS credentials file exists on the instance (e.g., /root/.aws/credentials or /home/{user}/.aws/credentials). If such a file exists, it might be interfering with the CodeDeploy agent’s ability to connect.

To fix this:

Delete the credentials file:

sudo rm -rf /root/.aws/credentials
# or for a specific user:
sudo rm -rf /home/{user}/.aws/credentials

Restart the CodeDeploy agent:

sudo systemctl restart codedeploy-agent

Conclusion

By following these steps, you should be able to resolve the "CodeDeploy cannot reach instance service" error and get your deployments back on track. If the issue persists, revisit the IAM role permissions and the CodeDeploy agent logs to gather more information.

References: