DEV Community: Lance Nehring

Git Worktrees: Replace Your Pile of Clones with One Manageable Repository

Lance Nehring — Sun, 26 Jul 2026 15:23:28 +0000

What is a Git worktree?
Why use worktrees instead of several clones?
A practical directory convention
Everyday Git worktree commands
Consolidating several independent clones
Safety rules before starting
Phase 1: Inventory every clone
Phase 2: Choose the canonical repository
Phase 3: Decide what each clone should become
Phase 4: Convert one clone
Moving a worktree
Recovering a deleted .git worktree file
When should a worktree be locked?
When should git worktree prune be used?
Final validation
Quick reference
Closing thoughts

Have you ever ended up with a directory structure like this?

~/src/project
~/src2/project
~/src3/project
~/src4/project

Each directory started innocently enough.

One was for main. Another was for a feature branch. A third contained a half-finished experiment. The fourth had several untracked test files you were afraid to lose.

Eventually, each clone had its own:

stale view of the remote repository,
duplicated Git history,
local-only commits,
modified files,
ignored test artifacts,
and unknown relationship to the others.

Git worktrees are designed to solve this problem.

A worktree gives you multiple checked-out working directories backed by one shared Git repository. Each working directory can have its own branch and uncommitted changes, while commits, branches, tags, remotes, and fetched objects remain shared.

This article covers two things:

How to use Git worktrees during normal development.
How to safely consolidate several independent clones into one worktree-based layout without losing local work.

The shell examples are written to work in both Bash and zsh.

What is a Git worktree?

A normal Git clone contains:

the object database,
commit history,
branches,
tags,
remotes,
remote-tracking references,
and one checked-out working directory.

A linked worktree adds another checked-out working directory to that same repository.

For example:

~/src/project                         main
~/src/project-FEATURE-123             FEATURE-123
~/src/project-HOTFIX-456              HOTFIX-456
~/src/project-large-restructure       large-restructure

Each worktree has its own:

checked-out branch or detached HEAD,
modified files,
untracked files,
ignored files,
staging area,
merge state,
and rebase state.

The worktrees share:

commit objects,
local branches,
tags,
remotes,
remote-tracking references,
and the results of git fetch.

That distinction is important.

A worktree is not merely another directory pointing at the same files. It is an independent working area backed by shared repository metadata.

Why use worktrees instead of several clones?

Commits are immediately visible everywhere

Suppose you commit something in a feature worktree:

git -C "$HOME/src/project-FEATURE-123" commit \
    -am "Complete FEATURE-123"

The commit and updated branch are immediately visible from the main worktree:

git -C "$HOME/src/project" log FEATURE-123 -1

There is no local push-and-fetch cycle between directories.

One fetch updates every worktree

Run:

git -C "$HOME/src/project" fetch origin --prune

Every linked worktree immediately sees the updated origin/* references.

With independent clones, every clone must be fetched separately, and each clone can have a different idea of what origin/main means.

Git prevents one branch from being checked out twice

Git normally refuses to check out the same local branch in two linked worktrees at once.

That protects you from independently modifying one branch from two directories.

Repository history is not duplicated

Each worktree has a separate checked-out filesystem tree, but the object database and Git history are shared.

For a large repository, that can save substantial disk space.

You get a central inventory

Run this from any worktree:

git worktree list

Example:

/home/user/src/project                    81f9d2a [main]
/home/user/src/project-FEATURE-123        f27ae91 [FEATURE-123]
/home/user/src/project-HOTFIX-456         6b812b0 [HOTFIX-456]

For script-friendly output:

git worktree list --porcelain

A practical directory convention

Git does not impose a directory naming convention, but this one is easy to understand:

~/src/<repository>                 main
~/src/<repository>-<branch>        another branch

Examples:

~/src/project
~/src/project-FEATURE-123
~/src/project-HOTFIX-456

Branch names containing / should usually be converted to filesystem-friendly names:

feature/new-api  ->  project-feature-new-api

The worktree directory name does not need to match the branch name exactly.

Everyday Git worktree commands

Before getting into migration, here are the commands you will use during normal development.

List worktrees

git worktree list

Add an existing local branch

git worktree add \
    ../project-FEATURE-123 \
    FEATURE-123

Create a new branch and worktree

git worktree add \
    -b FEATURE-123 \
    ../project-FEATURE-123 \
    main

This creates FEATURE-123 from main and checks it out in the new directory.

Create a worktree from a remote branch

Update the remote-tracking references:

git fetch origin --prune

Then create the local branch and worktree:

git worktree add \
    -b FEATURE-123 \
    ../project-FEATURE-123 \
    origin/FEATURE-123

Create a detached worktree for temporary testing

git worktree add \
    --detach \
    ../project-test \
    main

A detached worktree is useful for:

running a build against an old commit,
reviewing a release tag,
reproducing a bug,
or performing disposable testing.

Use a named branch instead if you may want to retain commits.

Work normally inside a worktree

Once created, a worktree behaves like an ordinary Git working directory:

cd ../project-FEATURE-123

git status
git add .
git commit
git pull --rebase
git push

You do not need special worktree versions of ordinary Git commands.

Rebase a feature branch onto updated main

Update the main worktree:

git -C ../project fetch origin --prune

git -C ../project merge \
    --ff-only \
    origin/main

Then rebase the feature worktree:

git -C ../project-FEATURE-123 rebase main

Because the worktrees share local branches, the feature worktree immediately sees the updated main.

Remove a worktree

git worktree remove ../project-FEATURE-123

Git normally refuses to remove a worktree containing uncommitted changes.

This is much safer than deleting the directory manually.

Consolidating several independent clones

Suppose you currently have:

~/src/project
~/src2/project
~/src3/project
~/src4/project

The target layout is:

~/src/project                         canonical repository and main
~/src/project-FEATURE-123             linked worktree
~/src/project-HOTFIX-456              linked worktree
~/src/project-large-restructure       linked worktree

The safe approach is:

Inventory every clone.
Choose one canonical repository.
Update the canonical repository carefully.
Convert one clone at a time.
Import the source clone's exact branch and commit.
Rename the source clone to a backup.
Create a linked worktree.
Copy the complete working-directory state.
Compare the old clone and new worktree.
Delete the backup only after testing.

Do not try to automate the entire migration in one opaque script.

Convert one work area, verify it, and then continue.

Safety rules before starting

A few rules prevent most migration disasters.

Keep the source clone as a backup

Do not delete an old clone during migration.

Rename it:

mv "$source_repo" "$backup"

Only delete the backup after the new worktree has been compared, built, and tested.

Do not use `exit` in pasted interactive snippets

A command such as:

exit 1

is reasonable inside a standalone script.

It is not friendly in a command block intended to be pasted into an interactive terminal because it may close the shell session.

The examples in this article print errors and rely on the engineer to stop before continuing.

Use braced variables in Bash and zsh

When text immediately follows a variable, use:

"${branch}:refs/heads/${branch}"

rather than:

"$branch:refs/heads/$branch"

The braced form is valid in both Bash and zsh.

This matters particularly when a variable is followed by :. Zsh can interpret the colon as part of parameter-expansion syntax and produce a malformed Git refspec.

Remember that a linked worktree has a `.git` file

A normal clone has:

.git/

A linked worktree usually has:

.git

That .git entry is a text file pointing back to the shared repository metadata.

This difference becomes important when using rsync.

Phase 1: Inventory every clone

Define the known clone paths:

repos=(
    "$HOME/src/project"
    "$HOME/src2/project"
    "$HOME/src3/project"
    "$HOME/src4/project"
)

This array syntax works in both Bash and zsh.

Inventory each clone:

for repo in "${repos[@]}"; do
    echo
    echo "================================================================"
    echo "Repository: $repo"
    echo "================================================================"

    if ! git -C "$repo" rev-parse --git-dir >/dev/null 2>&1; then
        echo "NOT A GIT REPOSITORY"
        continue
    fi

    printf "Branch:       "
    git -C "$repo" branch --show-current

    printf "HEAD:         "
    git -C "$repo" rev-parse HEAD

    printf "Origin:       "
    git -C "$repo" remote get-url origin 2>/dev/null ||
        echo "(none)"

    printf "Upstream:     "
    git -C "$repo" rev-parse \
        --abbrev-ref \
        --symbolic-full-name '@{upstream}' 2>/dev/null ||
        echo "(none)"

    echo
    echo "Status:"
    git -C "$repo" status \
        --short \
        --branch \
        --untracked-files=all

    echo
    echo "Recent commits:"
    git -C "$repo" log \
        --oneline \
        --decorate \
        -5
done

For each clone, record:

directory,
checked-out branch,
exact HEAD commit,
origin URL,
upstream branch,
modified files,
deleted files,
untracked files,
and local-only commits.

Do not forget ignored files

Normal git status hides ignored files.

Check them separately:

for repo in "${repos[@]}"; do
    echo
    echo "===== $repo: ignored files ====="

    git -C "$repo" status \
        --short \
        --ignored=matching \
        --untracked-files=all |
        grep '^!!' || true
done

Ignored files can still be important.

Examples include:

local configuration,
test credentials,
IDE settings,
generated test fixtures,
build caches,
and environment-specific inventory files.

You need to decide whether each one should be preserved.

Phase 2: Choose the canonical repository

Normally, the clone already used for main becomes the canonical repository:

canonical="$HOME/src/project"

Fetch the current remote state:

git -C "$canonical" fetch origin --prune

Inspect local and remote main:

git -C "$canonical" status --short --branch

git -C "$canonical" log \
    --oneline \
    --decorate \
    --graph \
    -10 \
    main origin/main

Check for untracked-file collisions

Before fast-forwarding, inspect incoming paths:

git -C "$canonical" diff \
    --name-status \
    HEAD..origin/main

A future tracked file may collide with a local untracked file at the same path.

Git will normally refuse to overwrite it, but the safer workflow is to preserve the local file explicitly.

pre_update_backup="$HOME/project-pre-update-$(date +%Y%m%d-%H%M%S)"

mkdir -p "$pre_update_backup/path/to"

mv "$canonical/path/to/local-file.yml" \
   "$pre_update_backup/path/to/local-file.yml"

Then fast-forward only:

git -C "$canonical" merge \
    --ff-only \
    origin/main

Compare the saved local file with the newly tracked version before restoring anything.

Phase 3: Decide what each clone should become

Clone on a different branch with useful work

Convert it into a linked worktree.

Another clone of `main`

Git normally permits a local branch to be checked out in only one worktree.

If the duplicate clone is clean and contains nothing unique, retire it.

Check for local-only commits:

git -C "$source_repo" log \
    --oneline \
    origin/main..main

Check tracked, untracked, and ignored files:

git -C "$source_repo" status \
    --short \
    --ignored=matching \
    --untracked-files=all

If the duplicate main clone is being used as a second test environment, create either:

a local alias branch, or
a detached worktree.

Local alias example:

git -C "$canonical" branch \
    local/second-main-testing \
    main

git -C "$canonical" worktree add \
    "$HOME/src/project-second-main-testing" \
    local/second-main-testing

Detached example:

git -C "$canonical" worktree add \
    --detach \
    "$HOME/src/project-main-test" \
    main

Use a named branch if commits may need to be retained.

Phase 4: Convert one clone

The following example converts one source clone into one linked worktree.

Set migration variables

canonical="$HOME/src/project"
source_repo="$HOME/src2/project"
branch="FEATURE-123"
destination="$HOME/src/project-FEATURE-123"
backup="$HOME/src2/project.pre-worktree-$(date +%Y%m%d-%H%M%S)"
migration_ref="refs/remotes/migrate/${branch}"

Verify the source:

git -C "$source_repo" branch --show-current
git -C "$source_repo" rev-parse HEAD

git -C "$source_repo" status \
    --short \
    --branch \
    --untracked-files=all

If git branch --show-current prints nothing, the source clone is in a detached HEAD state. Investigate before continuing unless a detached result is intentional.

Import the exact source branch

Do not assume the branch in the source clone matches:

the canonical repository's local branch,
origin/<branch>,
or the latest remote commit.

Import the source clone's exact branch into a temporary reference:

git -C "$canonical" fetch \
    "$source_repo" \
    "refs/heads/${branch}:${migration_ref}"

This also imports local-only commits that have never been pushed.

If the local branch does not already exist in the canonical repository, create it:

if git -C "$canonical" show-ref \
    --verify \
    --quiet "refs/heads/${branch}"
then
    echo "Local branch already exists:"
    git -C "$canonical" rev-parse "$branch"
else
    echo "Creating local branch from imported source"
    git -C "$canonical" branch \
        "$branch" \
        "$migration_ref"
fi

Now compare the two branch tips:

printf 'Source clone:  '
git -C "$source_repo" rev-parse HEAD

printf 'Canonical:     '
git -C "$canonical" rev-parse "$branch"

The hashes must be identical before continuing.

Do not blindly reset either repository if they differ. Investigate why the branch histories are different.

After verification, remove the temporary migration reference:

git -C "$canonical" update-ref \
    -d \
    "$migration_ref"

Rename the old clone

echo "Backup path: $backup"

mv "$source_repo" "$backup"

Confirm that the backup is still a valid Git repository:

git -C "$backup" rev-parse HEAD

git -C "$backup" status \
    --short \
    --branch \
    --untracked-files=all

Create the linked worktree

git -C "$canonical" worktree add \
    "$destination" \
    "$branch"

Compare the starting commits:

printf 'Old clone:     '
git -C "$backup" rev-parse HEAD

printf 'New worktree:  '
git -C "$destination" rev-parse HEAD

Again, the hashes must match.

Do not copy the old working files onto a worktree based on a different commit. The resulting diff would mix branch differences with uncommitted working-directory changes.

Restore the complete working-directory state

Use rsync to reproduce:

modified tracked files,
deleted tracked files,
untracked files,
ignored files,
and local generated files.

rsync -a \
    --delete \
    --exclude='/.git' \
    "$backup/" \
    "$destination/"

The anchored exclusion is critical:

--exclude='/.git'

Do not use only:

--exclude='.git/'

The trailing slash pattern protects a .git directory, but a linked worktree has a .git file. With --delete, the wrong exclusion can delete that file and temporarily break the worktree.

Restore upstream tracking

If the branch exists on origin, restore its upstream relationship:

if git -C "$canonical" show-ref \
    --verify \
    --quiet "refs/remotes/origin/${branch}"
then
    git -C "$destination" branch \
        --set-upstream-to="origin/${branch}" \
        "$branch"
else
    echo "No origin/${branch}; branch remains local-only"
fi

Compare Git-visible state

Capture the old and new status:

git -C "$backup" \
    status --porcelain=v1 --untracked-files=all \
    > /tmp/project-old.status

git -C "$destination" \
    status --porcelain=v1 --untracked-files=all \
    > /tmp/project-new.status

Compare them:

diff -u \
    /tmp/project-old.status \
    /tmp/project-new.status

No output means Git sees the same:

modified files,
deleted files,
and untracked files.

Compare the complete filesystem

Use a checksum-based dry run to include ignored files:

rsync -a \
    --checksum \
    --dry-run \
    --delete \
    --exclude='/.git' \
    "$backup/" \
    "$destination/"

No output means the filesystems match, excluding Git metadata.

At this point, build and test the new worktree before deleting the backup.

Moving a worktree

Same-filesystem move

Use Git's worktree-aware move command:

git -C "$canonical" worktree move \
    "$old_path" \
    "$new_path"

Cross-device move

A move between filesystems may fail with an error similar to:

Cross-device link

This often happens when moving between:

internal storage and an external disk,
two mounted volumes,
or separate filesystem partitions.

Move the directory using the operating system:

mv "$old_path" "$new_path"

Then repair Git's recorded paths:

git -C "$canonical" worktree repair \
    "$new_path"

Verify the result:

git -C "$canonical" worktree list

git -C "$new_path" rev-parse HEAD

git -C "$new_path" status \
    --short \
    --branch

file "$new_path/.git"
cat "$new_path/.git"

The .git file should contain a pointer resembling:

gitdir: /path/to/canonical/.git/worktrees/project-FEATURE-123

Do not rely only on the messages printed by git worktree repair. The final verification commands determine whether the repair succeeded.

Recovering a deleted `.git` worktree file

If an rsync --delete operation removed the linked worktree's .git file, Git commands in that directory may report:

fatal: not a git repository

Repair it from the canonical repository:

git -C "$canonical" worktree repair \
    "$destination"

Then verify:

file "$destination/.git"
cat "$destination/.git"

git -C "$destination" rev-parse HEAD

git -C "$destination" status \
    --short \
    --branch \
    --untracked-files=all

For future copies, use:

--exclude='/.git'

When should a worktree be locked?

Locking is useful when:

the canonical repository remains available,
one linked worktree lives elsewhere,
and that linked worktree may temporarily disappear.

Example:

~/src/project                         canonical repository on internal storage
/Volumes/External/project-testing     linked worktree on removable storage

Lock the removable worktree:

git worktree lock \
    --reason "Stored on removable volume" \
    /Volumes/External/project-testing

This tells Git not to treat the missing worktree as an abandoned entry eligible for pruning.

Unlock it before intentionally moving or removing it:

git worktree unlock \
    /Volumes/External/project-testing

When locking is unnecessary

Suppose the canonical repository and every linked worktree are together on the same external drive:

/Volumes/External/src/project
/Volumes/External/src/project-FEATURE-123
/Volumes/External/src/project-HOTFIX-456

When the drive is disconnected, the entire repository is unavailable. Git cannot run maintenance against it.

Locking every worktree does not provide meaningful additional protection in that arrangement.

Reconnect the drive at the same mount path and continue working normally.

When should `git worktree prune` be used?

Git stores administrative metadata for each linked worktree in the canonical repository.

If a worktree directory is removed outside Git, its administrative record may remain behind.

Examples:

Someone ran rm -rf on a worktree directory.
A temporary build worktree was deleted manually.
A storage volume was permanently retired.
A cross-device migration left an obsolete worktree record.

Preview stale records first:

git worktree prune \
    --dry-run \
    --verbose

Only after confirming that every listed worktree is permanently gone:

git worktree prune

prune removes obsolete administrative records. It does not delete an existing, accessible worktree directory.

Do not prune simply because a removable drive is temporarily disconnected.

Instead:

reconnect the drive, or
lock the removable linked worktree if the canonical repository remains available elsewhere.

Final validation

List the registered worktrees:

git -C "$canonical" worktree list

Check each one:

for worktree in \
    "$HOME/src/project" \
    "$HOME/src/project-FEATURE-123" \
    "$HOME/src/project-HOTFIX-456"
do
    echo
    echo "===== $worktree ====="

    git -C "$worktree" status \
        --short \
        --branch \
        --untracked-files=all
done

Inspect the shared and per-worktree metadata:

git -C "$destination" rev-parse \
    --show-toplevel

git -C "$destination" rev-parse \
    --git-common-dir

git -C "$destination" rev-parse \
    --git-dir

Build and test every converted worktree.

Only then remove its backup:

rm -rf "$backup"

Check the path carefully before running that command.

Quick reference

# List worktrees
git worktree list

# Machine-readable list
git worktree list --porcelain

# Add an existing local branch
git worktree add ../project-FEATURE FEATURE

# Create a new branch and worktree
git worktree add \
    -b FEATURE \
    ../project-FEATURE \
    main

# Create a detached testing worktree
git worktree add \
    --detach \
    ../project-test \
    main

# Move a worktree on the same filesystem
git worktree move ../old-location ../new-location

# Repair after a manual or cross-device move
git worktree repair ../new-location

# Lock one removable worktree while the canonical repo remains available
git worktree lock \
    --reason "Removable storage" \
    ../project-FEATURE

# Unlock before intentionally moving or removing it
git worktree unlock ../project-FEATURE

# Remove a registered worktree
git worktree remove ../project-FEATURE

# Preview obsolete worktree records
git worktree prune --dry-run --verbose

# Remove confirmed-obsolete records
git worktree prune

Closing thoughts

Git worktrees are not an exotic Git feature reserved for unusual workflows.

They are useful whenever you need to:

maintain several active branches,
compare old and new releases,
run concurrent builds,
preserve a long-running experiment,
review another branch without disturbing current work,
or replace a growing collection of independent clones.

The safest migration strategy is intentionally conservative:

preserve the source clone,
import its exact commit,
create the worktree at that commit,
copy the working-directory state,
compare everything,
test it,
and only then remove the backup.

Once the migration is complete, everyday work becomes simpler:

one fetch,
one repository history,
one set of remotes,
and as many independent working directories as you actually need.

Adventures with K0S in AWS

Lance Nehring — Fri, 09 Aug 2024 14:22:47 +0000

At the time of this writing (August 2024), K0S is at version v1.30.3. There's a tremendous about of outdated and incorrect information on the Internet (which impacts AI, if you're into asking AmazonQ or ChatGPT questions), so be aware of the date of this article. My goal is to keep it current - we'll see how that goes.

This isn't actually a tutorial - the end state is not desirable and the information is too dense. This is a more of an "engineering notebook" - akin to what my fellow graybeards may recall from engineering school.

My plan was to establish a Kubernetes presence on AWS without the incurring the costs of Amazon's EKS. I wanted a lightweight, but fully functional, K8S installation that I could stand up and tear down to prove out orchestration and deployment of containerized projects that come along.... such as those for a startup company where attention to cloud cost is paramount. I'm certainly not against EKS for those situations where the cost is justified and I have used it heavily in the past.

Picking through the various smaller K8S projects out there, I've settled on K0S since it's supposed to be "The Zero Friction Kubernetes". The features I'm after with this experiment are similar to what I've used with EKS:

ability to pull images from ECR
use the AWS cloud provider functionality to get those AWS specific things: using tags to annotate subnets, worker nodes, route tables, etc for use by the K0S installation.
use the pod identity agent to address pods that require certain privileges within AWS via IAM roles
use an ingress controller to manage the provisioning and lifecycle of AWS ELBs - namely NLBs and ALBs. K0S has a tool called "k0sctl" to manage installation, but it requires SSH access to the nodes. I have no other use for SSH and don't need to expand the attack surface, so I won't install it.

Establish a VPC for testing

I won't cover the mechanics of creating the VPC, subnets, Internet gateway, route table, security groups, NACLs, etc. I personally use the IaC tool Terraform whenever possible. There is a learning curve to use something like Terraform (and learning HCL), but the benefits are enormous - especially when you need consistency so you don't waste time chasing ghosts resulting from misconfigured infrastructure.

I'm using a VPC with a class B private CIDR (172.16.0.0/16) in the us-east-1 region, enabled for DNS hostnames and DNS resolution. I created 3 public subnets (each with a 20 bit subnet mask) even though we're only using 1 subnet to start with. The main route table needs a route for 0.0.0.0/0 that goes to the Internet Gateway for the VPC. I didn't create any private subnets in order to reduce the cost and need for any NAT gateways for this experiment.

Create your EC2 instance

Similarly, I won't cover the mechanics of creating an EC2 instance. Terraform comes in really handy here since you may find yourself repeatedly doing terraform apply and terraform destroy as you start and stop your experiments. I'm using a "t3a.large" node to start with using the latest AL2023 AMI - to have enough vCPU, memory, and networking to keep us out of harms way, without costing too much (in case we forget to destroy the instance after testing). Also, I'm not bothering to set up SSH to get a shell on the instance and I'm using AWS System Manager instead.

Establish an IAM role for the EC2 instance to use as its instance profile

We're going to do ourselves a giant favor in terms of security and start using IAM roles immediately. Many articles related to AWS just talk about putting credentials in some "~/.aws/credentials" file. Yes, you can do that, but you immediately create an issue that will fail a security audit, and you're actually making your life harder by having to track and secure those credentials. So don't cheat and use your personal IAM access keys, or Krampus will find you.
You can use Terraform for this as well. Effectively you need an IAM role, I named mine "k0s_instance" and attached these AWS managed policies:

AmazonEC2ContainerRegistryReadOnly
AmazonEKS_CNI_Policy (for when we experiment with the AWS CNI) You'll also need to create 2 customer managed policies and attach those to the role. The policy permissions information is from the docs here: https://cloud-provider-aws.sigs.k8s.io/prerequisites/#iam-policies I named mine "k0s_control_plane_policy" and "k0s_node_policy".

Install k0s

We're going to start at the smallest cluster - a single node. All of the following installation steps are done in a "root" shell on the EC2 instance. This means the control plane and worker artifacts will be running on the same node. There are side effects with node selection and tolerations that we'll run into, but we'll address that later.
https://docs.k0sproject.io/stable/install/

curl -sSLf https://get.k0s.sh | sudo sh
k0s sysinfo
mkdir -p /etc/k0s
k0s config create > /etc/k0s/k0s.yaml

Get the ECR credential provider binary

This link can help you determine what releases are available: https://github.com/kubernetes/cloud-provider-aws/releases

The AWS credential provider documentation is very light. You get to create a configuration file and then search the K0S docs for how to manipulate the kubelet arguments to use that file. This article is for K3S, but shows that the configuration file can be YAML instead of JSON - something that isn't mentioned in the credential provider docs.

RELEASE=v1.30.3
curl -OL https://storage.googleapis.com/k8s-staging-provider-aws/releases/${RELEASE}/linux/amd64/ecr-credential-provider-linux-amd64
mv ecr-credential-provider-linux-amd64 /etc/k0s/ecr-credential-provider
chmod 0755 /etc/k0s/ecr-credential-provider
cat << EOF > /etc/k0s/custom-credential-providers.yaml
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
- name: ecr-credential-provider
  matchImages:
  - "*.dkr.ecr.*.amazonaws.com"
  - "*.dkr.ecr.*.amazonaws.com.cn"
  apiVersion: credentialprovider.kubelet.k8s.io/v1
  defaultCacheDuration: '0'
EOF

Edit the k0s.yaml

Here is where we actually set the kubelet argments so that the ECR credential provider will work. The clues for this were from this Gist. Combining that information with this K0S doc, we discover that it is possible to use "--kubelet-extra-args" on the k0s command line to set the "--extra-args for the kubelet.
Also, there seems like no possible way to get the default K0S CNI of kuberouter to work in AWS. I don't know the root cause - possibly there's CIDR conflicts with what I chose for my VPC CIDR - but it was a simple change to set the "spec.network.provider" value to "calico" in the "/etc/k0s/k0s.yaml" file that we created. Calico worked fine for me without further configuration.
So, for now we're using Calico as the CNI. I feel like I should be able to use the AWS VPC CNI plugin, but that has not yet been successful for me. This may need to be revisited if the AWS Load Balancer Controller requires it.

k0s install controller --single --enable-cloud-provider --kubelet-extra-args="--image-credential-provider-config=/etc/k0s/custom-credential-providers.yaml --image-credential-provider-bin-dir=/etc/k0s" -c /etc/k0s/k0s.yaml
systemctl daemon-reload
k0s start

This will "start" the cluster, which we need to do to get the our kubectl configured that we'll do next. The single node cluster won't truly start yet - and that's ok for now. You'll notice Pods in the pending state.

k0s status
k0s kubectl get pod -A

Install and configure kubectl

Here we grab a version of kubectl that matches our kubernetes version so that we maximize compatibility. We use the k0s command to generate a valid config file and put it in the expected place. Note the this file contains the "keys to the kingdom" as far as the k0s installation is concerned, so treat it appropriately.

k0s version
curl -LO https://dl.k8s.io/release/v1.30.3/bin/linux/amd64/kubectl
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
mkdir -p ~/.kube
k0s kubeconfig admin > ~/.kube/config
chmod 0600 ~/.kube/config

Install helm and add the stable and cloud-provider-aws repos

We're embracing helm charts for repeatable, stable, versioned installations of everything we can. Install the latest version of helm and setup a few repos that we intend to use.

dnf install -y git
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add stable https://charts.helm.sh/stable
helm repo add aws-cloud-controller-manager https://kubernetes.github.io/cloud-provider-aws
helm repo add eks https://aws.github.io/eks-charts
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Install the helm chart for the aws-cloud-controller-manager

AWS Tags

The documentation for the AWS Cloud Provider is rather underwhelming. Especially frustrating is the lack of direct information about tagging AWS resources. There's some information here that can help:

Using a kubernetes cluster name of "testcluster", the tags we start with are:

Tags for EC2 instance, VPC, subnets:

Key	Value
kubernetes.io/cluster/testcluster	owned

Additional tags for subnets:

Key	Value
kubernetes.io/role/elb	1
kubernetes.io/role/alb-ingress	1
kubernetes.io/role/internal-elb	1

AWS Cloud Provider

I found it necessary to edit the node selector and tolerations of the daemonset to be able to get the pod scheduled in the case of this single node deployment. I was also unable to get AWS route tables annotated to the point where the aws-cloud-controller-manager would be happy about configuring cloud routes. Not sure what "cloud routes" are supposed to be, but for now, I've disabled that feature. There's more on it here.
We are doing all this in the custom helm values file.

cat << EOF > /etc/k0s/accm-values.yaml
---
args:
  - --v=2
  - --cloud-provider=aws
  - --configure-cloud-routes=false
nodeSelector:
  node-role.kubernetes.io/control-plane: "true"
tolerations:
- key: node.cloudprovider.kubernetes.io/uninitialized
  value: "true"
  effect: NoSchedule
EOF
helm -n kube-system upgrade --install aws-cloud-controller-manager aws-cloud-controller-manager/aws-cloud-controller-manager --values /etc/k0s/accm-values.yaml

Install pod identity agent

Unfortunately, I didn't see a helm chart for the eks-pod-identity-agent hosted on a Helm repo. So we're forced to clone the git repo and install the helm chart from that work area.

cat << EOF > /etc/k0s/epia-values.yaml
---
clusterName: testcluster
env:
  AWS_REGION: us-east-1
EOF
git clone https://github.com/aws/eks-pod-identity-agent.git
cd eks-pod-identity-agent/
helm install eks-pod-identity-agent --namespace kube-system ./charts/eks-pod-identity-agent --values ./charts/eks-pod-identity-agent/values.yaml --values /etc/k0s/epia-values.yaml

Is it working so far?

Kubectl should be happy with the node and the pods. It can take a few minutes for the pods to reach a "Running" state.

kubectl get node -o wide
kubectl get pod -A -o wide

From what I've seen, the worker node (our only node, at this point) must have IP addresses assigned, else the kubectl logs command will fail if you try to inspect logs from the pods/containers. I found that you can find logs in the "/var/log/container" container directory of the EC2 instance that I'm using.

Ingress Controller

Here's where things take another complicated twist. The AWS Cloud Controller Manager contains a legacy AWS load balancer controller capable of managing legacy ELBs and NLBs. The code for the NLB management shows an older and newer API... I'm not sure what the switch is between the two, but it may very well be the "--v=2" argument that was passed to the aws-cloud-controller-manager. Oddly the newer API for NLBs is not capable of configuring for proxy protocol, whereas the docs suggest that it does, and so does the code for the older API.
It appears that this legacy code in the aws-cloud-controller-manager is bascially EOL - you can still use it, but broken things are not getting fixed. The push seems to be with a follow-on project, the AWS Load Balancer Controller. It is absolutely confusing, but I did find an article to explain it better.

Install the Nginx Ingress Controller

https://kubernetes.github.io/ingress-nginx/
The available customization values can be found with this nifty helm command:

helm show values ingress-nginx --repo https://kubernetes.github.io/ingress-nginx

We're creating a special configuration here based on: https://kubernetes.github.io/ingress-nginx/deploy/#network-load-balancer-nlb

The idea is that the Nginx ingress controller is behind an NLB that accepts HTTPS and HTTP traffic (TCP:443 and TCP:80). The HTTPS traffic has the SSL terminated at the NLB using the certificate given the the annotation. That traffic of HTTPS origin, is then fed to the nginx controller as HTTP traffic. The traffic of HTTP origin is sent by the NLB to a "tohttps" port (TCP:2443) at the nginx controller, that merely responds to the client with a code 308 permanent redirect - to force the client to use HTTPS.

Notes:

The "$" in the http-snippet are escaped in this heredoc to protect them from the shell.
The shell variable CERT_ARN must be set to whatever certificate ARN that you have in your AWS Certificate Manager that you intend to use.
Since this annotation using the legacy AWS load balancer controller, only a single certificate ARN can be specified.
The "proxy-real-ip-cidr" is set to the CIDR of the VPC I'm using. You can force proxy protocol to work, by uncommenting the comments in the heredoc. The controller will not actually enable proxy protocol on the NLB's target groups, so you'll have to use the AWS console and do that manually. It can work, but it's not solution for production.

CERT_ARN="xxxxxxxx"
cat << EOF > /etc/k0s/nic-values.yaml
---
controller:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ${CERT_ARN}
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
#      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    targetPorts:
      http: tohttps
      https: http
  config:
#    use-proxy-protocol: "true"
    use-forwarded-headers: "true"
    proxy-real-ip-cidr: "172.16.0.0/16"
    http-snippet: |
      server {
        listen 2443;
        return 308 https://\$host\$request_uri;
      }
  containerPort:
    tohttps: 2443
EOF
helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx --values /etc/k0s/nic-values.yaml -n ingress-nginx --create-namespace

Simple application for testing

Once the nginx ingress controller is running, we can attempt to test. For simplicity, this application is just a yaml manifest instead of a helm chart (there's likely a better way to do this). You can adjust the ingress host to something other than "web.example.com" - to potentially match that SSL cert that you're using - or not, depending on whether your testing can handle SSL name mismatch errors.

"simple-web-server-with-ingress.yaml":

apiVersion: v1
kind: Namespace
metadata:
  name: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
  namespace: web
spec:
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: httpd
        image: httpd:2.4.53-alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web-server-service
  namespace: web
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-server-ingress
  namespace: web
spec:
  ingressClassName: nginx
  rules:
  - host: web.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-server-service
            port:
              number: 5000

Apply with kubectl apply -f simple-web-server-with-ingress.yaml. It will take a few minutes for the NLB to finish provisioning and pass initial health checks. You can monitor the progress in the AWS EC2 console. The ingress and service can be seen with:

kubectl get ingress -A -o wide
kubectl get service -A -o wide

Now you can attempt to curl to the NLB's DNS address (same as the address shown by kubectl). I'm giving curl the "-k" option to ignore the SSL cert mismatch, and I'm also setting a "Host" HTTP header, since the ingress is explicitly for "web.example.com".

So, when I execute:

curl -k -H 'Host: web.example.com' https://xxxxxxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com

I get the expected:

<html><body><h1>It works!</h1></body></html>

If I try to similarly curl using HTTP instead of HTTPS, as:

curl -k -H 'Host: web.example.com' http://xxxxxxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com

Then I get the expected 308 permanent redirect:

<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

Note that since I didn't change "web.example.com" to some DNS that I own, that if I tell curl to follow the redirect as:

curl -L -k -H 'Host: web.example.com' http://xxxxxxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com

I get the expected error:

curl: (6) Could not resolve host: web.example.com

Conclusions at this point

We've shown that it is possible to get K0S working on a single node in AWS. Using the nginx ingress controller can work for an NLB, but there are issues that make it undesirable:

It's using a legacy AWS load balancer controller contained in the AWS cloud manager controller, which means:
- Risk of that code being removed at some unknown point in the future
- Current documentation doesn't match the actual features
- NLBs are not configurable for TLS SNI or proxy protocol
- No support for ALBs
The nginx ingress controller has a somewhat complicated configuration to accomplish a common HTTP to HTTPS redirect.

Moving forward

In the current AWS Load Balancer Controller docs we find this:

Additional requirements for non-EKS clusters:

Ensure subnets are tagged appropriately for auto-discovery to work

For IP targets, pods must have IPs from the VPC subnets. You can configure the amazon-vpc-cni-k8s plugin for this purpose.

I'm going to revisit using the amazon-vpc-cni-k8s plugin. I'm thinking that I missed the kubelet configuration requirements when experimenting before and didn't actually have it installed properly. It appears that they may be components that require installation directly on the worker node - like with the ECR credential provider. We'll see - every day is a learning experience.

Has anyone else tried to use K0S in this way? or have advice/clarifications/questions that I may (or may not) be able to answer?