Debugging Node.js applications running on ECS Fargate

AWS Fargate is a serverless compute engine for containers. It removes the need to provision and manage servers. To know more on Fargate, you can visit this page.
In this blog post, we will see how to debug a Node.js application running on ECS Fargate using the new ECS Exec feature. Once this is done, you'll be able to debug your application using Chrome DevTools. I used this to debug memory leaks in a Node.js application.

Prerequisites

• SSH and Google Chrome installed (to create SSH tunnel).
• A SSH key pair (public key and private key). I used PuTTYgen to generate and save these keys.
• AWS CLI configured.
• AWS Systems Manager Session Manager plugin installed.
• Node.js application running in ECS Fargate.
• ECS service's security group has ports 22 and 9229 open in inbound rules.

We'll use SSH tunnel to establish connection between the Fargate server and the local system.

AWS Configurations

Considering that you have a task already running in ECS Fargate, the next step is to enable debugging for the application.

Enable Debugging Mode

When enabled, the Node.js process listens for a debugging client on port 9229. To enable this, you have to add --inspect when running the application. Example:

node server.js --inspect

Enable ECS Exec Feature

Open the ECS Task role in IAM and add below policy to the role:

{
   "Version": "2012-10-17",
   "Statement": [
       {
       "Effect": "Allow",
       "Action": [
            "ssmmessages:CreateControlChannel",
            "ssmmessages:CreateDataChannel",
            "ssmmessages:OpenControlChannel",
            "ssmmessages:OpenDataChannel"
       ],
      "Resource": "*"
      }
   ]
}

Next, we run the AWS CLI command:

aws ecs update-service --service <service-name> --cluster <cluster-name> \
    --region <region> \
    --enable-execute-command --force-new-deployment

Replace the content inside angular brackets with appropriate values.

Once the above command is executed, a new task will start with ECS Exec enabled. If you receive any error when running the above command, please check the error response you received and try to resolve it. You can find troubleshooting tips from AWS docs.

Install requirements in the ECS task

Now we can ECS Exec into the running task and configure SSH. Use the below command to exec into the ECS task:

aws ecs execute-command --cluster <cluster-name> \
    --task <task-id> \
    --container <container-name-to-execute-the-command-on> \
    --interactive \
    --command "/bin/sh"

Now you should be getting an interactive session in the Fargate server. Next, run:

apt-get update
apt-get install openssh-server -y
service ssh start

Now you'll have SSH server running in the ECS task.
We'll have to do a few more steps to get this working right. Create a file called docker-entrypoint.sh and paste below code to it:

#!/bin/sh
SSH_PUBLIC_KEY="<ssh-public-key>"
if [ -z "$SSH_PUBLIC_KEY" ]; then
  echo "Need your SSH public key"
  exit 1
fi
# Create a folder to store user's SSH keys if it does not exist.
USER_SSH_KEYS_FOLDER=~/.ssh
[ ! -d "$USER_SSH_KEYS_FOLDER" ] && mkdir -p $USER_SSH_KEYS_FOLDER

# Copy contents from the `SSH_PUBLIC_KEY` variable
# to the `${USER_SSH_KEYS_FOLDER}/authorized_keys` file.
echo $SSH_PUBLIC_KEY > ${USER_SSH_KEYS_FOLDER}/authorized_keys

# Clear the `SSH_PUBLIC_KEY` variable.
unset SSH_PUBLIC_KEY

# Start the SSH daemon.
/usr/sbin/sshd -D
echo "Success"

Replace <ssh-public-key> with the SSH public key you generated earlier. Now save the file.
So far so good. Next, run the script:

chmod +x docker-entrypoint.sh 
./docker-entrypoint.sh 

Now we come back to our local system.

Local System Configurations

Go to the directory where the SSH private key is stored and run:

chmod 700 ssh

To test the SSH connection, you can run:

ssh root@<ecs-task-ip> -i ssh

Finally run the command to create the SSH tunnel:

ssh -nNT -L 9229:127.0.0.1:9229 root@<ecs-task-ip> -i ssh

That should do it!
Now open Chrome and enter chrome://inspect in the URL. DevTools page will appear. Notice the "Remote Target" section. Below it, you should be able to find the Fargate application for debugging.

Thanks for reading!

References

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html

If you like this post, please consider buying me a coffee.

Comments

[Emmanuel K]

Great article. Helped me a lot.

However, you don't need SSH (no need to install ssh server or certifcate). You only need to open up port 9229 and then you can run the following (after checking that ecs exec works correctly):

aws ssm start-session \
--target "ecs:CLUSTER-NAME_TASK-ID_CONTAINER-RUNTIME-ID" \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["9229"], "localPortNumber":["9229"]}' \
--profile AWS_PROFILE

And then open inspector in chrome and you should see your container: