Troubleshooting Docker credsStore Auto-Configuration Issues in VS Code Dev Containers

When using VS Code's Dev Container extension, you might encounter credentials-related errors while trying to access registries through docker pull, az acr login, or other CLI tools. These errors might look like this:

error getting credentials - err: exit status 255, out: ``

You might also experience failures with commands like kcl mod add or az acr login, or encounter 403 errors related to authentication. This article explains the cause and solutions to these issues.

Note about "Host Machine" in this article

In Dev Containers, the "host machine" refers to the machine running VS Code.

Even if your Docker daemon is running on a cloud or remote server and you're only running VS Code on your local Mac or PC, we'll refer to the VS Code side (local PC) = host machine in this context.

The Problem

• Commands like kcl mod add or docker pull succeed when executed from VS Code's terminal inside the container, but fail with error getting credentials when accessing the container through SSH or docker exec.
• Looking at ~/.docker/config.json inside the container, you'll find credsStore is set to something like dev-containers-xxxxxx....
• Removing this credsStore temporarily fixes the error, but it reappears when reconnecting to the Dev Container in VS Code.

Root Cause

Understanding Docker Credential Helper and VS Code Dev Containers Extension

VS Code's Dev Container extension includes a feature that automatically sets up a Docker Credential Helper. This mechanism attempts to share registry authentication information with the container using the ~/.docker/config.json and credsStore from the host machine running VS Code, not from within the container.

1. When starting a Dev Container, if there's no credsStore specified in the container's ~/.docker/config.json, and Dev Container settings are default (or dev.containers.dockerCredentialHelper is enabled), the VS Code Dev Containers extension automatically inserts a credsStore.
2. This results in the following content in the container's ~/.docker/config.json:

{
  "credsStore": "dev-containers-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

1. Consequently, when running docker pull or az acr login inside the container, it attempts to call a helper like docker-credential-dev-containers-xxxxxx.
   • This helper is a generated script inside the container that performs IPC (Inter-Process Communication) with VS Code (Dev Container extension) to query authentication information tied to the host machine's ~/.docker/config.json.

The REMOTE_CONTAINERS_IPC Environment Variable

The success of this authentication depends on the REMOTE_CONTAINERS_IPC environment variable.

• When attaching to a container through VS Code's terminal, the Dev Container extension automatically sets REMOTE_CONTAINERS_IPC, allowing successful authentication via credsStore for commands like docker pull.
• However, when accessing the container through SSH or docker exec without VS Code, this environment variable isn't set, causing the docker-credential-xxx to fail its IPC communication, resulting in the "error getting credentials - err: exit status 255" error.

Solutions

1. Remove credsStore from the Container's ~/.docker/config.json

Opening ~/.docker/config.json inside the container and removing the credsStore line will temporarily allow docker pull and similar commands to succeed.
Example:

# Inside container: ~/.docker/config.json
{
  // "credsStore": "dev-containers-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  "auths": { ... }
}

However, this isn't a permanent solution if VS Code settings remain default, as credsStore will be re-added on each startup.

2. Disable dev.containers.dockerCredentialHelper

In your Dev Container configuration file (devcontainer.json), you can prevent the Docker Credential Helper from being installed by setting dev.containers.dockerCredentialHelper: false:

// devcontainer.json example
{
  "customizations": {
    "vscode": {
      "settings": {
        // Set this to false
        "dev.containers.dockerCredentialHelper": false
      }
    }
  }
}

This prevents automatic credsStore configuration in the container's ~/.docker/config.json, eliminating IPC errors. However, you'll lose the ability to inherit Docker authentication from the host machine, requiring separate docker login commands in the container and consideration of associated security risks.

3. Enable REMOTE_CONTAINERS_IPC for SSH/Alternative Shells

If you want to maintain IPC functionality even when accessing the container without VS Code, you can persist the REMOTE_CONTAINERS_IPC value set by VS Code across shells.

For example, using Dev Container's postAttachCommand, you could write the environment variable to ~/.config/fish/conf.d/ (for fish shell):

if [ -n "${REMOTE_CONTAINERS_IPC+x}" ]; then
  # Export environment variable for fish shell
  echo "set -x REMOTE_CONTAINERS_IPC ${REMOTE_CONTAINERS_IPC}" > ~/.config/fish/conf.d/remote-containers-ipc.fish
fi

This allows recognition of REMOTE_CONTAINERS_IPC even when accessing via SSH. However, since REMOTE_CONTAINERS_IPC might have random values per connection, you'll need to establish operational rules.

4. Clean Up Host Machine Credentials

Authentication issues like 403 errors during docker pull might occur if credential information in the host machine's ~/.docker/config.json or key management (like Keychain Access) is corrupted.

• On macOS, try deleting credentials for ghcr.io, quay.io, etc., from Keychain Access and then run docker login again.
• Verify that tokens obtained after docker login have necessary permissions like packages:read.

Reviewing these aspects can help prevent 403 errors (insufficient permissions).

Summary

• VS Code's Dev Container extension includes a Docker Credential Helper that inherits Docker authentication from the host machine.
  • Here, "host machine" means the machine running VS Code, not the Docker daemon host.
• This mechanism automatically configures credsStore in the container's ~/.docker/config.json, but can fail authentication in environments without REMOTE_CONTAINERS_IPC (like SSH).
• Available solutions include:
  1. Removing credsStore from the container's ~/.docker/config.json (though it may be re-added)
  2. Disabling the Credential Helper by setting dev.containers.dockerCredentialHelper to false
  3. Implementing a custom mechanism to persist REMOTE_CONTAINERS_IPC across external shells
  4. Cleaning up Docker credentials on the host machine (checking/re-logging via Keychain Access)

References

• Devcontainer version 0.275 - auto adding ~/.docker/config.json with credsStore specified which cause az acr login to fail · Issue #7982 · microsoft/vscode-remote-release