What happens when your migration agent dies at file 7 of 12

A migration agent spends real money and real minutes per run. It clones a repo, installs its deps, plans a file order, then rewrites files one at a time, running the target's test suite after each one. On a mid-sized repo that is a lot of Bedrock calls and a lot of wall-clock time.

So the question I had to answer early: what happens when the process dies at file 7 of 12?

If the answer is "start over," the agent is a toy. You have already paid for six migrated files and a passing test suite on each, and a spot reclamation or a redeploy throws all of it away. Worse, the API side runs on Lambda, which has a hard 15-minute ceiling. A real migration blows past that. So the long-horizon work cannot live where the request comes in, and the thing doing the work has to survive being killed.

Here is how RepoModernizer handles it, and the bug that resume introduced that I did not see coming.

The split: API on Lambda, worker on Fargate

The API is a FastAPI app behind API Gateway, running on Lambda through Mangum. It takes the request, writes nothing important, and drops a message on SQS. That is it. It never runs the migration.

An SQS consumer Lambda reads the message and calls ecs.run_task to launch a one-shot Fargate task. The Fargate task is the worker. It runs the LangGraph pipeline to completion (or until it dies) and then exits. Nothing stays warm. Idle cost is zero because there is nothing idle.

# consumer_handler.py — Lambda that turns an SQS message into a Fargate run
_ecs.run_task(
    cluster=os.environ["ECS_CLUSTER"],
    taskDefinition=os.environ["ECS_TASK_DEFINITION"],
    launchType="FARGATE",
    overrides={
        "containerOverrides": [{
            "name": "worker",
            "environment": [
                {"name": str(k).upper(), "value": str(v)}
                for k, v in body.items() if v is not None
            ],
        }]
    },
)

The whole reason the worker is on Fargate and not Lambda is that 15-minute cap. Lambda is fine for the request. It is not fine for a job that might run half an hour.

Two kinds of durable, in two different places

The worker has two things it cannot afford to lose when it dies: the repo it is editing, and the graph state that says how far it got. These live in two different stores, on purpose.

The repo lives on EFS, mounted into the Fargate task at /mnt/workspace. The task clones the target repo there, makes a branch, and every diff it applies lands on that filesystem. When a new task picks up the same job, the files are already sitting there in the state the last run left them.

The graph state lives in DynamoDB. Every node the LangGraph pipeline finishes gets written as a checkpoint. That is what lets a fresh process know it already ingested, already installed deps, already migrated files 0 through 6, and should resume at file 7.

Splitting them matters. DynamoDB holds the plan, the cursor, and the per-file status. EFS holds the actual bytes. Neither one alone is enough to resume. You need the state to know where you were and the filesystem to have the work you already did.

The checkpointer is a real LangGraph saver, not a side table

LangGraph has a checkpointer interface, BaseCheckpointSaver. The out-of-the-box options are an in-memory saver (useless across processes) and integrations you may or may not want to run. I wrote one backed by DynamoDB directly, because DynamoDB was already in the stack and I wanted single-table access patterns I controlled.

The core is put and get_tuple. put serializes the checkpoint with LangGraph's own JsonPlusSerializer and writes it under a per-task partition key:

def put(self, config, checkpoint, metadata, new_versions):
    thread_id = config["configurable"]["thread_id"]
    checkpoint_id = checkpoint["id"]
    parent_id = config["configurable"].get("checkpoint_id")
    self._put_blob(
        f"TASK#{thread_id}", f"CKPT#{checkpoint_id}", (checkpoint, metadata),
        extra={"parent_checkpoint_id": parent_id or ""},
    )
    return {"configurable": {
        "thread_id": thread_id,
        "checkpoint_ns": config["configurable"].get("checkpoint_ns", ""),
        "checkpoint_id": checkpoint_id,
    }}

get_tuple does the reverse. Given a task id and no specific checkpoint, it queries the latest one:

resp = self._table.query(
    KeyConditionExpression=Key("PK").eq(f"TASK#{thread_id}")
        & Key("SK").begins_with("CKPT#"),
    ScanIndexForward=False, Limit=1,
)

ScanIndexForward=False, Limit=1 gives the most recent checkpoint for that task. On cold start the worker builds the graph with this checkpointer attached, and LangGraph replays from the last saved node. Every checkpoint carries a 14-day TTL so old task state ages out of the table on its own.

The graph build is where the checkpointer gets wired in:

graph = build_graph(deps, checkpointer=checkpointer)
config = {"configurable": {"thread_id": task_id}}

thread_id is the task id. That single value ties a resuming process back to everything the previous process wrote.

What "resume" actually does, and what it does not

I want to be precise here, because it is easy to oversell this.

Resume works at the node level, not the byte level inside a file. If the worker dies while migrating file 7, it does not continue from the middle of file 7. On the next run, the loop restarts the migrate_file node for the current cursor from the top: it re-reads the file off EFS, asks the model for fresh content, applies the diff, and reruns tests. Files 0 through 6 are already marked done in the checkpointed state, so it does not touch them. It picks up at the current cursor and moves forward.

So "resumes mid-migration" is true in the sense that matters (you do not redo six files of completed, test-passing work), and false in the sense that a single interrupted file gets redone from scratch. That is the honest version.

The bug: resume, done twice, opens the same PR six times

The pipeline has a human-in-the-loop gate. When the planner scores a file's risk above a threshold, the migrate_file node calls LangGraph's interrupt() with the diff, and the run pauses. A human approves or rejects. Approval comes back in as a separate action that resumes the graph with Command(resume=...).

Here is the thing I missed. That migrate_file node re-executes from its top on every resume. Re-reads the model, re-applies, re-runs tests. That is fine when there is genuinely a pending interrupt to satisfy. It is not fine when the resume action fires twice.

And it fires twice more often than you would think. SQS can deliver a message more than once. A user can click approve again while a slow Fargate cold start has not yet updated the checkpoint, so the UI still shows a pending decision. Either way, a second resume lands for a task whose interrupt was already cleared by the first one.

Calling invoke() again does not no-op. It runs the node fresh, produces a new diff, applies it, and drives the run all the way to finalize again, which opens a pull request. I found this live: one approve click produced six to nine identical commits on the PR.

The first fix was to check whether there is actually a pending interrupt before resuming:

snapshot = graph.get_state(config)
has_pending_interrupt = any(t.interrupts for t in snapshot.tasks)

That helped. It did not fix it. Two approve calls sent about a second apart still produced one extra commit, not zero. The reason is that get_state then invoke is check-then-act. Two callers landing in the same instant both read has_pending_interrupt = True before either one resumes. Classic race.

The real fix is atomicity, and DynamoDB is the only thing in this system that offers it. I added a try_claim that does a conditional put: the write only succeeds if the item does not already exist.

def try_claim(self, task_id: str, key: str) -> bool:
    try:
        self._table.put_item(
            Item={"PK": f"TASK#{task_id}", "SK": f"CLAIM#{key}",
                  "ttl": int(time.time()) + _TTL_SECONDS},
            ConditionExpression="attribute_not_exists(PK)",
        )
        return True
    except ClientError as exc:
        if exc.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

The key is the checkpoint id being resumed. Only the first caller to claim that specific checkpoint proceeds to invoke. The second gets False and returns the existing state without re-running anything.

claimed = checkpointer.try_claim(task_id, f"resume:{checkpoint_id}")
if has_pending_interrupt and claimed:
    result = graph.invoke(Command(resume={...}), config=config)
else:
    result = snapshot.values

There is a second guard on the far side too, at finalize: before opening a PR, check whether this task already has a PR URL recorded, and bail if it does. Belt and suspenders, because the cost of a duplicate PR is annoying and public.

The other thing that broke: git, EFS, and root

Two smaller landmines, both worth knowing if you mount EFS into a container that runs git.

The EFS access point pins every file to uid 1000 no matter who writes it. The container runs as root. Git 2.35.2 and up refuses to operate on a repo it sees as owned by a different user (that is the CVE-2022-24765 dubious-ownership check), so the first git command after the clone blows up. Fix is one line at worker startup:

subprocess.run(["git", "config", "--global", "--add", "safe.directory", "*"], check=True)

The second one showed up only on JS repos. The diff step normalizes the target file to end in a newline before diffing. git apply matches context against real bytes, and difflib never emits the "No newline at end of file" marker, so a file without a trailing newline fails to apply cleanly. Python source almost always ends in a newline, so my Python fixtures never caught it. Hand-written JS and JSON often do not, and it broke against a real JS repo.

What I would do differently

The duplicate-commit bug is the interesting one, and the lesson is not "add a claim." The lesson is that I built idempotency in the wrong layer first. I reached for a state check (has_pending_interrupt) when the problem was concurrency, and a state check cannot solve concurrency. Only a conditional write can. If I were doing it again I would put the atomic claim in from the start, at every resume boundary, and treat the state check as a cheap early-out rather than the guarantee.

The other thing: migrate_file re-running from the top on resume is convenient but wasteful. It re-pays for a Bedrock call on a file it may have already generated. A cleaner design caches the generated content against the checkpoint so a resume that is really just a duplicate delivery costs nothing. I have not built that yet.

Durability is not one feature. It is durable state, durable storage, and idempotent resume, and the last one is the one that bit me.