Troubleshooting AWS Cloud9 - RuntimeError: Container does not exist / No space left on device

Issue description

Recently I got a runtime error in AWS Cloud9 while trying to build my serverless application as a Docker container image. I used AWS SAM build command:

sam build --use-container

As a result I got the following error issue:

RuntimeError: Container does not exist. Cannot get logs for this container

It turned out, AWS tried to create amazon/aws-sam-cli-build-image-python3.8 Docker container image:

but failed:

Then I tried to pull the image manually using docker command:

docker pull amazon/aws-sam-cli-build-image-python3.8

but it looks like there is not enough disk space:

You can use the following command to verify that the root volume mounted under "/" is full:

df -h

Here is an output with 95% out of 10 GiB of EBS storage (/dev/xvda1) is used:

To check whether the EBS volume has a partition that must be extended, use the following command:

lsblk

It displays information about the block devices attached to your instance.

Solution

Once the problem is understood, improvement comes naturally. All I need is to increase the size of my EBS volume.

👉 Step 1. Open a new file in Cloud9

👉 Step 2. Add the script below to the file:

#!/bin/bash

# Specify the desired volume size in GiB as a command-line argument. If not specified, default to 20 GiB.
SIZE=${1:-20}

# Get the ID of the environment host Amazon EC2 instance.
INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)

# Get the ID of the Amazon EBS volume associated with the instance.
VOLUMEID=$(aws ec2 describe-instances \
  --instance-id $INSTANCEID \
  --query "Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId" \
  --output text)

# Resize the EBS volume.
aws ec2 modify-volume --volume-id $VOLUMEID --size $SIZE

# Wait for the resize to finish.
while [ \
  "$(aws ec2 describe-volumes-modifications \
    --volume-id $VOLUMEID \
    --filters Name=modification-state,Values="optimizing","completed" \
    --query "length(VolumesModifications)"\
    --output text)" != "1" ]; do
sleep 1
done

#Check if we're on an NVMe filesystem
if [ $(readlink -f /dev/xvda) = "/dev/xvda" ]
then
  # Rewrite the partition table so that the partition takes up all the space that it can.
  sudo growpart /dev/xvda 1

  # Expand the size of the file system.
  # Check if we are on AL2
  STR=$(cat /etc/os-release)
  SUB="VERSION_ID=\"2\""
  if [[ "$STR" == *"$SUB"* ]]
  then
    sudo xfs_growfs -d /
  else
    sudo resize2fs /dev/xvda1
  fi

else
  # Rewrite the partition table so that the partition takes up all the space that it can.
  sudo growpart /dev/nvme0n1 1

  # Expand the size of the file system.
  # Check if we're on AL2
  STR=$(cat /etc/os-release)
  SUB="VERSION_ID=\"2\""
  if [[ "$STR" == *"$SUB"* ]]
  then
    sudo xfs_growfs -d /
  else
    sudo resize2fs /dev/nvme0n1p1
  fi
fi 

👉 Step 3. Save the file as resize.sh (press Ctrl + S)

👉 Step 4. Run sh-file (shell script):

sh resize.sh 20

You should get detailed information about data block changes:

Here is the link to AWS instuction.

Results

To verify a new EBS volume size, run df -h command again:

Once there is enough disk space for Docker, my container issue is resolved:

Hopefully, this solution could save you both some time and headaches.