How to debug containers running in an AWS Beanstalk?

The beginning

One upon a time, a team who have their infrastructure in a Cloud provider, AWS. They created many of AWS resources like Route53, S3 buckets, IAM users, roles, policies ... & AWS Beanstalk environments.
The team dedicated their monitoring to an external company/service and was happy 'till they needs to add one more container in an AWS Beanstalk environment (and of course they will need to debug inside them what is happening, it's the reason of this story ^^).

Have you ever struggle with AWS Benstalk?
Me? Yes, it's my case! :-D

AWS = Russian dolls

In my point of view, I imagine AWS as "russian dolls".
Let's take a concrete example. I need to deploy an AWS Beanstalk (EB) with containers, OK so I should create a Beanstalk environment, a Beanstalk application and a Beanstalk application versions, linked to a ZIP file I need to host in a S3 bucket containing my application code and a Dockerrun.aws.json file (it's a docker-compose like for AWS).

And concretely, when we will create a Beanstalk environment, AWS will also create an ECS (Elastic Container Service) in an EC2 (a compute instance = a virtual machine).
So we have:

Beanstalk (EB) > ECS > EC2 > our containers

It's our russian dolls! :-D

Benstalk = Magic wand?

Like I have the habits to repeat things again and again, there is no "magic wand" technology.

In order to understand how is it working, you need to know all the things/resources created by AWS when you click on a button :-).

When everything is working well we are generally happy but what happens when we have to debug a service which is managed, which launches other services and which looks like a black box from the outside?

Personally I like to know how things work and when I had to deal with an AWS Beanstalk, which was running containers, me who is used to playing with Kubernetes and Docker, to debug my containers directly, my first question to summer: OK that's nice, it's a managed service but where are my containers, where are they running, how can i access them and how to debug them?

Debugging

Let's dig in the concrete part of this article: how debug containers in an AWS Beanstalk environment?

Like we said, we created an AWS Beanstalk application version (with the application code), linked to an AWS Beanstalk application link to an AWS Beanstalk environment. Beanstalk will create ECS tasks and everything you configured in the Beanstalk environment will run in fact in an AWS EC2 instance.

Aahh everything is running in an EC2 instance, it's an interesting fact. So, how debug our containers running in this EC2 instance?

Fortunately you can do it with one useful tool and some tricks!

EB-CLI

The first interesting tool when you want to debug a Beanstalk is the EB-CLI.

With this CLI you can do a lot of things:

• create an environment
• execute a health check to it
• watch EB logs
• launch an EB locally
• terminate an EB
• ...
• and connect in SSH to an AWS EB! :-)

FYI, the good practice is to use AWS Session manager instead of SSH in order to connect to EC2 instances.

Connect to Beanstalk

$ eb ssh

This command will allows us to connect to an environment. The first thing you need to do is to select the wanted Beanstalk environment.

/!\ If you configured an EC2 key pair, in the Beanstalk env, you need to add it in your .ssh/ folder in your local machine.

Ok, so let's execute eb ssh command:

$ eb ssh
INFO: Running ssh -i /Users/aurelievache/.ssh/toto.pem [e-mail]
Enter passphrase for key '/Users/aurelievache/.ssh/toto.pem':
Last login: Fri Jan 17 14:22:17 2021 from xxxx-xx-xx-xxx.wxx-xx.xx.xxx.fr
 _____ _           _   _      ____                       _        _ _
| ____| | __ _ ___| |_(_) ___| __ )  ___  __ _ _ __  ___| |_ __ _| | | __
|  _| | |/ _{% raw %}` / __| __| |/ __|  _ \ / _ \/ _` | '_ \/ __| __/ _` | | |/ /
| |___| | (_| \__ \ |_| | (__| |_) |  __/ (_| | | | \__ \ || (_| | |   <
|_____|_|\__,_|___/\__|_|\___|____/ \___|\__,_|_| |_|___/\__\__,_|_|_|\_\
                                       Amazon Linux AMI

This EC2 instance is managed by AWS Elastic Beanstalk. Changes made via SSH
WILL BE LOST if the instance is replaced by auto-scaling. For more information
on customizing your Elastic Beanstalk environment, see our documentation here:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
[ec2-user@ip-xx-x-x-xxx ~]$ 
```

Cool! I am finally in the EC2 where my containers are running!
It's not longer a blackbox and now I can use my favorite tools :-D.

## Let's debug our applications!

So, it's easy, let's see the states of my docker containers:

```
$ docker ps

ERROR: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
```

What???
Docker is not installed/running? :'( 
I can't debug my containers? :-(

Don't panic, in fact, Docker is running, but, there is a trick... you need to switch to root user!

```
[ec2-user@ip-xx-x-x-xxx ~]$ sudo su -

[root@ip-xx-x-x-xxx ~]# docker ps

CONTAINER ID        IMAGE                                                     COMMAND                  CREATED              STATUS                   PORTS                                            NAMES
a6b7fa123445        nginx:1.19                                                "/docker-entrypoint.…"   About a minute ago   Up About a minute        0.0.0.0:80->80/tcp                               ecs-myeb-toto-gfshfqhsgj-41-nginx-container-8ae78cdf789fdf7877
...
```

Ahhhhh, I can see my loving & running containers.

Knowing the Container ID of our running containers, now I can watch logs:

`[root@ip-xx-x-x-xxx ~]# docker logs <container-ID>`

And I can execute commands directly into it in order to debug them (and understand what is happening!):

`[root@ip-xx-x-x-xxx ~]# docker exec -it <container-ID> bash`

For example if I want to understand why my mysql-container doesn't have the correct database and tables into it, I can execute useful commands:

```
[root@ip-xx-x-x-xxx ~]# docker exec -it <container-ID> bash
> ls -l
> ls -l "docker-entrypoint-initdb.​d/"
> head "docker-entrypoint-initdb.​d/db-init.sql"
CREATE DATABASE myAwesomeDB;
> ls -l var/lib/mysql
# ...
```        

Perfect!

If you want some tips about [debugging and troubleshooting in the Docker context](https://dev.to/aurelievache/understanding-docker-part-22-debugging-troubleshooting-docker-5dbn), you can see my sketchnote :-).

# Conclusion

Don't think your managed resources as blackbox but try to understand what is happening when you click in a button in your favorite Cloud provider or when you deploy a resource through Infrastructure as Code (IaC) tool like Terraform or CloudFormation.

It's very useful to know how this not-magic world is working :-).