Worktree lifecycle for AI agents

The problem is not creating worktrees, but managing them

When two or more agents work in parallel on different features, a problem
appears that Git does not solve by itself: how to manage changes so the agents
do not interfere with one another.

One possible strategy is agent-to-agent communication. Agents can delegate
tasks, transfer context, declare ownership, and perform handoffs. This helps
coordinate intent and dependencies, but it is not physical isolation. If one
agent has to wait for another agent's response, contract, or commit before it
can continue, the supposed parallelism can become a sequence of coordinated
steps.

Claude Code's documentation makes this distinction explicit: agent teams can
communicate and share tasks, while worktrees are used when changes to files
need to be isolated.

Worktrees complement that strategy. Each agent works in its own working tree
and can modify files without directly overwriting another agent's checkout.
This reduces interference during editing, but it does not eliminate semantic
conflicts: two agents can still design incompatible changes or depend on
different interfaces.

Communication does not disappear either. It may still be necessary to
coordinate a contract, a commit, a merge, or a later integration.

That raises another question: what happens after a worktree is created, and
how is its lifecycle managed?

An isolated worktree has no lifecycle by itself

Git provides worktrees, but it does not define how their entire lifecycle
should be managed.

A first management layer can be found in the
obra/superpowers/using-git-worktrees
skill. It detects whether the agent is already inside a worktree, distinguishes
a submodule from a worktree, selects a location, verifies that the directory
is ignored, creates the worktree, prepares dependencies, and runs a test
baseline.

That flow solves provisioning and initial setup. Operational questions still
remain: how is the work integrated? What happens when a merge fails? How do we
avoid removing a branch with an unexpected state? What happens if the process
dies during shutdown? How are local changes in the destination recovered?

That is the space worktree.sh attempts to cover: not replacing Git's
worktree creation, but adding a lifecycle layer for agents that create, use,
integrate, and abandon work environments in an automated way.

worktree.sh: a lifecycle for worktrees

worktree.sh is a tool for managing the lifecycle of worktrees. Its
parent-only interface includes these operations:

• create: creates and validates the worktree and its working branch.
• complete: verifies the state, performs the squash merge, and removes resources when the closing conditions are met.
• cleanup: removes the worktree without integrating it; destructive discard requires --discard.
• recover: resumes pending preserve-dirty transactions.
• prune: cleans up Git's worktree records and reports unassociated directories.
• list: shows the worktrees registered by Git.

The tool also adds ID validation, base-branch selection, predictable paths and
branches, locks for incompatible operations, and journals that record closing
phases.

An agent's workflow

The workflow starts when the agent receives a work-id. From that ID,
worktree.sh creates a separate branch and working tree from the detected base
branch or from the branch provided explicitly.

work request
    ↓
create worktree
    ↓
agent modifies isolated files
    ↓
prepare / verify, when configured
    ↓
complete or cleanup

A minimal way to start work is:

.flowtask/scripts/worktree.sh create hotfix/test-001 --base main

The agent works inside the worktree without directly modifying the shared
checkout. When it finishes, it can run configured validations and close the
work with:

.flowtask/scripts/worktree.sh complete hotfix/test-001 --base main

complete verifies the worktree's identity and state, integrates commits with
a squash merge, and only then removes the branch and worktree when the closing
validations succeed.

If the work must be abandoned without integration, the workflow uses
cleanup:

.flowtask/scripts/worktree.sh cleanup hotfix/test-001

Forced discard requires explicit authorization through --discard.

There is also a more delicate case: the destination may contain local changes
that should not become commits yet. With --preserve-dirty, the tool captures
eligible tracked and untracked changes, temporarily cleans the destination,
performs the squash merge, and then attempts to restore those changes.

Ignored files are not part of that preservation. If restoration encounters a
conflict, the transaction becomes pending_manual: the backup is preserved and
resolution requires human intervention. In that case, recover can resume the
transaction, but it does not resolve the conflict automatically.

What happens when something fails

The lifecycle must also define what happens if a process dies during a merge,
the worktree changes, or restoration finds a different state.

For that purpose, worktree.sh records closing phases in journals. A journal
identifies which steps have been confirmed and from where an operation can be
resumed. It is not merely an informational log: it is part of the state the
tool uses to decide whether it can continue or must stop.

For example, if complete attempts a squash merge and encounters a conflict,
the closing operation stops and preserves the worktree and branch. The tool
does not remove resources that may be needed to investigate or resolve the
conflict.

In a --preserve-dirty transaction, if the process dies after capturing the
destination changes but before restoring them, recover can resume the flow
from the recorded state.

If restoration encounters a conflict, the transaction enters pending_manual,
keeps the backup, and requires human intervention. recover can continue the
reconciliation after the state is corrected, but it does not decide how to
resolve the conflict.

This behavior is deliberate: when the state cannot be verified, the tool
prefers to stop and preserve resources rather than perform an irreversible
action.

The tool's current state

The current manager has a parent-only flow—the main repository creates and
closes its own worktrees—and a submodule flow.

Submodule initialization is explicit: it does not run by default when an agent
creates a worktree. This avoids populating dependencies the agent does not
need.

To act directly on a submodule, target-direct treats the submodule as the
target repository. The flow validates its relationship with the parent
repository, the path, the gitlink (the parent's reference to the submodule
commit), and branch identity.

link-parent is also available for controlled gitlink updates, with the
staged-only mode modifying only the index; it does not commit, merge, or push
the parent.

The tool does not yet expose an MCP interface or subtree integration. Evolution
toward MCP is already included in the project roadmap.

Project evolution

The manager's roadmap is organized into these stages:

1. Stabilize the standalone CLI
        ↓
2. Migrate the backend to Python
        ↓
3. Keep worktree.sh as a compatibility shim
        ↓
4. Expose the backend through MCP

The main direction is to separate the manager's core, preserve its guarantees
during the migration, and later provide an interface suitable for agents and
other orchestrators.

The code and the manager's evolution are available in the
worktree-manager repository.

Comments

[Auten]

The journal + pending_manual design is the right instinct: stopping with the backup intact beats a clever auto-resolve that loses someone's work.

A few edge cases worth covering as agents start calling this unattended:

• Base drift. The agent's validations ran against the base it branched from. If main moved since create, the squash merge can be clean in git terms and still broken. Running the configured checks again on the merged result before removing the worktree would catch that.
• git worktree lock with a reason (e.g. the work-id) while an agent is active, so a prune from another session or a cleanup script can't drop the metadata of a worktree that is mid-task.
• Ignored files cut both ways: .env, local DB files and dev-server ports aren't in the new worktree either, so two agents often collide on port 3000 or a shared SQLite file even with perfectly isolated source trees. A per-worktree port/env hook at create helps a lot.
• For the MCP stage: agents retry on timeouts, so making complete idempotent per work-id (second call reports the recorded result instead of merging again) and keeping --discard out of the default tool surface will save some bad days.

Does complete currently re-run validations after the squash, or only before it?

[Juan Saez]

Thanks for the detailed analysis. To answer your question: today complete runs validations only before the squash, so the base drift you describe is a real gap. On top of that, the squash is committed and there is no automatic rollback, so validating afterwards would leave a broken commit on the target.

I'll most likely solve it the other way around: store the base head at create and, in complete, if the base has advanced, integrate it into the worktree branch and validate there before touching the target. If there was no drift, the extra validation cost isn't paid. The lock with a reason and the per-work-id idempotency are on my list for the MCP stage.

One case I haven't solved yet: main can move again between that validation and the squash. Have you handled that window with a head re-check or with a broader lock?

The repo is public (Saez520/worktree-manager), so if you want to dig into any of these edge cases, an issue or a PR is welcome anytime.

[Auten]

Integrating the new base into the worktree branch and validating there is the right order: nothing touches the target until the result is known good.

For the last window I'd use a compare-and-swap rather than a broader lock. Build the squash commit without moving the target (commit-tree on the validated tree, parent = the base head you validated against), then move the ref with git update-ref refs/heads/main <squash> <validated-base-head>. That call is atomic and fails if main moved in between, so you either land exactly what you tested or nothing. On failure: re-integrate, re-validate, retry a bounded number of times, then stop with pending_manual like your conflict path.

A lock on top still helps, but only for serializing your own completes (one flock per repo, so two agents don't keep invalidating each other's validation). It can't stop a human push, the CAS covers that. If the target is a remote, a plain non-force push already rejects a moved base for the same reason.

[Juan Saez]

Follow-up with results, since I tested the CAS idea in isolated repos.

A raw update-ref CAS is atomic (with a stale expected value it fails with rc=128 and touches nothing), but on a checked-out branch it leaves the index and working tree behind: git status shows phantom staged changes. With --preserve-dirty, the capture verification then fails closed and cancels the close without modifying anything.

What did work: building the squash commit on the validated tree, with parent = the validated base head, and landing it with git merge --ff-only. It acts as the CAS (it rejects with rc=128 if main diverged and changes nothing), it updates the index and files, and in my tests it coexisted with --preserve-dirty.

I also measured the race itself: the tree I validate and the final squash tree are identical if the base doesn't move, and different if it does (an extra file ended up in the final commit that never passed validation).

One design note: complete doesn't run tests by design, since validation is done by the orchestrator above it. The real gap is that today it also doesn't flag when the base has changed since create, so the drift gets in silently. The direction I'm testing is for complete to detect it and return its own state ("base changed, revalidate"), with ff-only covering the final window.

Still untested: the full flow with a real validation step, and a crash between the ff-only and the journal write. Thanks, this made me rethink the close step.

[Auten]

Good catch on the phantom staged changes. update-ref only moves the ref, so ff-only is the better landing step whenever the target branch is checked out.

For the crash between ff-only and the journal write, I'd flip the order: write an intent record first (work-id, validated base, squash sha), then ff-only, then mark it done. On startup, any intent without done gets reconciled with git merge-base --is-ancestor <squash> main: if it's there, the land happened and you just close the record; if not, nothing landed and it's safe to retry. That also makes complete idempotent per work-id.

For "base changed, revalidate", returning both shas (base at create or last validation, current head) lets the orchestrator decide between a cheap re-run and a full one without asking git again.

[Reid Marlow]

The journal pattern solves the state machine, but the annoying edge case in automated agent runs is untracked build residue. When an agent times out or crashes during a test run, it usually leaves behind untracked artifacts like bytecode caches, coverage databases, or test logs. A strict cleanup fails because the tree is dirty, while a blind clean wipes out the exact crash logs you need to debug why the run failed. Archiving untracked telemetry to a separate run directory before pruning the tree saved me a lot of mystery failures.

[Juan Saez]

Thanks, this maps onto something I could test directly. Current behavior: complete and cleanup refuse to proceed when the worktree has untracked files (unless cleanup --discard), so a dirty tree after a crash is not silently wiped. But ignored files are a different story: they don't show up in the untracked check, so a normal cleanup removes the worktree with them inside. I confirmed both cases in an isolated test. Since bytecode caches and coverage databases are usually gitignored, the likely failure mode there is the opposite of the one you describe: not a blocked cleanup but silently lost evidence.

What survives a crash today is the worktree, the branch and the journal at the phase reached, so the evidence is there until someone cleans up. There is no archiving of test output or logs; only --preserve-dirty keeps transactional backups. The natural place for an archive step would be a phase between merged and deinit_started, before the worktree is removed.

Two things I'd need to decide: which patterns to archive (all untracked and ignored is probably too much, logs only may be too little), and how to cap size and avoid copying secrets like .env. What do you archive, and how do you handle retention?

[Auten]

Silently lost evidence is the worse failure, agreed, and the merged -> deinit_started gap is the right place for it.

On what to archive, I'd make it an explicit allowlist the repo declares (a small file next to your config), with a conservative default: test reports and logs (junit xml, .log, the coverage summary rather than the coverage database). An allowlist fails safe; "everything untracked minus a denylist" eventually copies something it shouldn't. Still keep a hard denylist on top that the repo can't override: .env, *.pem, *.key, anything under .ssh or credential stores.

For size, cap per file and per archive, and record what was skipped and why in the journal, so a missing log is visible rather than silent.

Retention: tie it to the outcome. A cleanly merged work-id keeps its archive a few days, failed or pending_manual ones keep it until someone resolves them, plus a global size ceiling that evicts the oldest successful ones first.