Docker-Compose Gettings IAM Error Credentials

Let's say you've successfully written your first docker-compose applications on your EC2 of your choice and voila deployed it! After maybe a few hours you might get errors like this:

Above was a streamlit project state that got deployed with docker-compose, but IAM credentials might happen regardless of your deployment configuration.

So what happened?

To understand this, we need a basic understanding of EC2 Instance Metadata.

EC2 instance metadata is a REST API accessible only from within the EC2 instance at a fixed link-local IP address: http://169.254.169.254/latest/meta-data/. It requires no authentication and provides information in plaintext or JSON format.

You can test this endpoint using:

TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/

So why is our Docker-Compose project failing then?

Put simply, its because usually we isolate the docker-compose projects into its own network bridge, for example:

services:
  api:
    build:
      context: .
      dockerfile: Dockerfile.api
    expose:
      - "8000"
    volumes:
      - ./tools:/app/tools
    environment:
      - PYTHONUNBUFFERED=1
    restart: unless-stopped
    networks:
      - app-network

  streamlit:
    build:
      context: .
      dockerfile: Dockerfile.streamlit
    ports:
      - "8501:8501"
    expose:
      - "8501"
    environment:
      - PYTHONUNBUFFERED=1
    depends_on:
      - api
    restart: unless-stopped
    networks:
      - app-network

networks:
  app-network:
    driver: bridge

When creating networks like above, accessing this metadata becomes challenging when applications run inside Docker containers, due to Docker’s network isolation and default networking configurations.

How do we solve it

The simplest way to grant a container access to EC2 metadata is to use Docker’s host network mode, which makes the container share the host’s network stack.

services:
  api:
    build:
      context: .
      dockerfile: Dockerfile.api
    volumes:
      - ./tools:/app/tools
    environment:
      - PYTHONUNBUFFERED=1
    restart: unless-stopped
    network_mode: host

  streamlit:
    build:
      context: .
      dockerfile: Dockerfile.streamlit
    environment:
      - PYTHONUNBUFFERED=1
      - API_SERVER_URL=http://api:8000
    depends_on:
      - api
    restart: unless-stopped
    network_mode: host

By sharing the network hosts port directly, this eliminates network isolation, allowing the container to directly access 169.254.169.254. Thus, we shouldn't have any issues regarding IAM service role credentials refreshing and EC2 instance metadata.