DEV Community

John Ajera
John Ajera

Posted on

Why Your Kustomize Remote Bases Break on Managed Argo CD (and How to Fix It)

#argocd #kubernetes #kustomize #vendoring

Why Your Kustomize Remote Bases Break on Managed Argo CD (and How to Fix It)

You switched to managed Argo CD (e.g. EKS Capabilities) and use AWS CodeConnections for Git. Your Applications that pull from public GitHub remotes in Kustomize suddenly fail with errors like "Password authentication is not supported" or "Authentication failed". The same kustomization worked with self-managed Argo CD. This article explains why CodeConnections causes that and how vendoring those remote bases into your repo fixes it.

1. Overview

What this guide does:

  • Explains why Kustomize remote resources (e.g. github.com/open-policy-agent/gatekeeper-library/...) break when Argo CD uses CodeConnections for Git
  • Shows that the credential helper is applied to every github.com URL, including public repos, and GitHub rejects those credentials
  • Describes when to vendor and a minimal how-to: download remote files at a pinned ref, add them to your repo, and point your kustomization at local files instead of remote URLs
  • Suggests keeping a README in the vendored directory so future upgrades are a repeatable re-download and commit

Why it breaks:

  • With managed Argo CD + CodeConnections, one credential is used for all GitHub access. When Kustomize runs git fetch for a remote base, Argo CD passes that same credential. Public repos don't need auth—but they receive it anyway, and GitHub rejects the format (e.g. "Password authentication is not supported").

2. Prerequisites

3. Why remote bases break with CodeConnections

When Argo CD syncs an Application, the repo-server runs kustomize build over your repo. If your kustomization.yaml lists remote resources like:

resources:
  - github.com/open-policy-agent/gatekeeper-library/library/general/httpsonly?ref=master
Enter fullscreen mode Exit fullscreen mode

Kustomize invokes git fetch for that URL. Argo CD configures a Git credential helper so that any git operation gets credentials. With CodeConnections, that helper returns the same credential (the one for your connected repos) for every github.com request—including requests to public repos such as open-policy-agent/gatekeeper-library.

Git sends those credentials to GitHub. GitHub then responds with something like:

fatal: Authentication failed for 'https://github.com/open-policy-agent/gatekeeper-library/'
remote: Invalid username or token. Password authentication is not supported for Git operations.
Enter fullscreen mode Exit fullscreen mode

So the build fails even though the remote repo is public and would work with no credentials. With self-managed Argo CD you might have used SSH for your app repo; Kustomize’s HTTPS fetches to public GitHub then didn’t use that credential and succeeded. With CodeConnections, one HTTPS credential is used everywhere, and it gets sent to public URLs where it’s invalid.

4. When to vendor

Vendor remote bases when:

  • You use managed Argo CD with CodeConnections and your Kustomize app references public GitHub remotes (e.g. gatekeeper-library, shared bases from other orgs).
  • You see authentication or "Password authentication is not supported" errors during manifest generation for those remotes.

You can also vendor remotes you don’t control so that upgrades are explicit and reproducible (pin to a commit, re-download when you want to upgrade).

5. How to vendor

Vendoring means copying the remote manifest files into your repo and pointing Kustomize at local files instead of remote URLs.

Steps:

  1. Choose a ref — Use a commit SHA or tag from the upstream repo (e.g. master or a specific SHA) so you can reproduce the same content later.
  2. Download the remote files — Each remote resource is usually a directory containing one or more YAML files (e.g. template.yaml). Download those files using the raw GitHub URL pattern: https://raw.githubusercontent.com/<org>/<repo>/<ref>/<path>/<file>.yaml Save them under a directory in your repo (e.g. infrastructure/my-app/vendored/) with clear names (e.g. httpsonly.yaml, requiredlabels.yaml).
  3. Update your kustomization — Replace remote resources entries with the local file names: 
# Before (remote – fails with CodeConnections)
resources:
  - github.com/open-policy-agent/gatekeeper-library/library/general/httpsonly?ref=master

# After (vendored)
resources:
  - httpsonly.yaml
Enter fullscreen mode Exit fullscreen mode
  1. Document the source and upgrade process — Add a README in the vendored directory that lists:
    • The upstream repo and the ref (commit SHA) you used
    • The mapping from each local file to its upstream path
    • How to upgrade: re-download from a new ref, overwrite the files, run kustomize build . to verify, commit, and update the README with the new ref.

Optionally, add a small script or one-liner (e.g. a shell loop with curl) that downloads all vendored files given a ref, so upgrades are a single command plus commit.

6. Summary

  • Problem: Managed Argo CD + CodeConnections sends one GitHub credential to every git URL. Kustomize remote bases to public GitHub repos get that credential; GitHub rejects it and the build fails.
  • Fix: Vendor those remote bases: download the manifest files at a pinned ref, put them in your repo, and point kustomization.yaml at the local files. No remote fetch at build time, so no credential is used for those resources.
  • Upgrades: Re-download from a new ref, overwrite the vendored files, verify with kustomize build ., commit, and update the README with the new ref.

Checklist when vendoring:

  1. Pick a ref (e.g. latest master or a commit SHA) from the upstream repo.
  2. Download each remote file via https://raw.githubusercontent.com/<org>/<repo>/<ref>/<path>/<file>.yaml into a directory in your repo.
  3. Replace remote resources in kustomization.yaml with the local file names.
  4. Add a README in that directory with source ref, file mapping, and upgrade steps (and optionally a download script).

7. Troubleshooting

Manifest generation still fails after vendoring

  • Confirm kustomization.yaml lists only local file names (e.g. httpsonly.yaml), not github.com/... URLs.
  • Run kustomize build . from the directory that contains the kustomization and vendored files; fix any path or resource errors before relying on Argo CD.

Upstream added or removed a file

  • Re-download from the ref you want (e.g. new commit on master). Add or remove the corresponding local file and update the resources list and README.

8. References

Top comments (0)