DEV Community

aimldl Account
aimldl Account

Posted on

Restoring a Vertex AI Training Cluster: Recovering a Stopped GPU Node from Expired Reservations

Restoring a Vertex AI Training Cluster: Recovering a Stopped GPU Node from Expired Reservations

Cluster Architecture

Google Cloud’s Vertex AI Training Cluster (VTC) is designed for large-scale AI/ML model training. The environment discussed here is a High-Performance Computing (HPC) cluster provisioned using VTC.

For example, the following screenshots show a cluster named vtctest001.

image

Click the name vtctest001to see the details.

image

image

image

The above cluster comprised of the following components:

Orchestration
Slurm Workload Manager is used to manage cluster resources and schedule jobs.

Compute Resources

  • Login Node is an N2-standard CPU instance acting as the cluster controller and entry point.
  • Compute Node is an a4-highgpu-8g instance equipped with 8x NVIDIA B200 GPUs.

Storage

  • Filestore is mounted at /home for persistent user directories.
  • Managed Lustre (Parallelstore) is mounted at /mnt/lustre for high-throughput training data access.

Networking
VPC (Virtual Private Cloud): a custom VPC vtc-test-net ensures low-latency communication between nodes.

Cloud Shell

Activate the Cloud Shell and run commands in the terminal.

image

For readability, the full prompt

admin_@cloudshell:~ (vertex-tkim)$
Enter fullscreen mode Exit fullscreen mode

is shown as

$
Enter fullscreen mode Exit fullscreen mode

Scenario

A GPU Virtual Machine (VM) vtctest001-a4-0 was originally provisioned using a reservation via DWS Calendar Mode. This reservation has since expired, causing the VM to enter a TERMINATED state.

$ gcloud compute instances list --project=$(gcloud config get-value project)
Enter fullscreen mode Exit fullscreen mode

(output example)

NAME: vtctest001-a4-0
ZONE: europe-west4-b
MACHINE_TYPE: a4-highgpu-8g
PREEMPTIBLE: 
INTERNAL_IP: 192.168.0.9
192.168.64.2
192.168.128.2
192.168.136.2
192.168.144.2
192.168.152.2
192.168.160.2
192.168.168.2
192.168.176.2
192.168.184.2
EXTERNAL_IP: 
STATUS: TERMINATED
  ...
$
Enter fullscreen mode Exit fullscreen mode

In Cloud Console, navigate to the Compute Engine > VM instances to obtain the same information:

image

Consequently, the cluster’s compute capacity is offline, effectively halting all training operations. Attempts to restart the VM fail because the specific reservation ID associated with the instance is no longer valid. Standard attempts to unbind the reservation also fail due to capacity restrictions on the machine type.

Approach to Resolve This

The node will be recovered using the DWS Flex-Start provisioning model.

Phase 1: Environment Configuration

Environment variables are defined to match the existing cluster configuration. This ensures that all subsequent commands target the specific resources within the project.

# Project and Zone Configuration
PROJECT_ID=$(gcloud config get-value project)
ZONE="europe-west4-b"

# Cluster Resource Names
GPU_NODE="vtctest001-a4-0"
LOGIN_NODE="vtctest001-login-001"
NETWORK="vtc-test-net"
Enter fullscreen mode Exit fullscreen mode

Phase 2: Safe Deletion of the VM Wrapper

The existing VM instance must be deleted to remove the binding to the expired reservation. Crucially, the --keep-disks=boot flag is required. This instruction destroys the compute instance (CPU/RAM allocation) but strictly preserves the boot disk (Storage) containing the operating system, Slurm configuration, and local data.

  1. Retrieve the boot disk name
BOOT_DISK_NAME=$(gcloud compute instances describe $GPU_NODE \
    --zone=$ZONE \
    --format="value(disks[0].source.basename())")
Enter fullscreen mode Exit fullscreen mode

Verify the variable

$ echo "BOOT_DISK_NAME=$BOOT_DISK_NAME"
vtctest001-a4-0
$
Enter fullscreen mode Exit fullscreen mode
  1. Create a snapshot of the boot disk
SNAPSHOT_NAME="snap-$GPU_NODE-recovery"

gcloud compute disks snapshot $BOOT_DISK_NAME \
    --snapshot-names=$SNAPSHOT_NAME \
    --zone=$ZONE
Enter fullscreen mode Exit fullscreen mode

Note: Detaching the disk from the stopped VM won’t work.

Creating snapshot(s) snap-vtctest001-a4-0-recovery...working      
Enter fullscreen mode Exit fullscreen mode

If this succeeds, your data is safe.

(output example)

Creating snapshot(s) snap-vtctest001-a4-0-recovery...done. 
$
Enter fullscreen mode Exit fullscreen mode

Phase 2: Force Delete the VM

Now that we have a backup, we can delete the VM without worrying about flags like --keep-disks (which was causing your error). We allow the original disk to be deleted with the VM.

gcloud compute instances delete $GPU_NODE \
    --zone=$ZONE
Enter fullscreen mode Exit fullscreen mode

(output example)

The following instances will be deleted. Any attached disks configured to be auto-deleted will be deleted unless they are attached to any other instances or the 
`--keep-disks` flag is given and specifies them for keeping. Deleting a disk is irreversible and any data on the disk will be lost.
 - [vtctest001-a4-0] in [europe-west4-b]

Do you want to continue (Y/n)?  
Enter fullscreen mode Exit fullscreen mode

Enter y

Do you want to continue (Y/n)?  y
Enter fullscreen mode Exit fullscreen mode
ERROR: (gcloud.compute.instances.delete) Could not fetch resource:
 - Required 'Current principal doesn't have permission to mutate this resource!' permission for 'vtctest001-a4-0'$
$
Enter fullscreen mode Exit fullscreen mode

Delete a faulty node

Node deletion is the default behavior if there’s no action specified. Whether the deleted node will be recreated depends on if the node is a dynamic or static node, controlled by Slurm.
If a static node is deleted (node index less than node pool scaling spec min node count), a new compute node with the same name and spec should be recreated.
If a dynamic node is deleted (node index larger than node pool scaling spec min node count), it won’t be recreated if there’s no workload scheduled on it.
To delete a faulty node:

gcurl -X POST  https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/modelDevelopmentClusters/CLUSTER_ID:reportFaultyNodes -d '{"nodeActions": [{"nodeId": "NODE_ID"}]}'
Enter fullscreen mode Exit fullscreen mode

NODE_ID is in a format of --
For example,

ENDPOINT="https://${REGION}-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/${REGION}/modelDevelopmentClusters"
CLUSTER_URI="${ENDPOINT}/${CLUSTER_ID}"

gcurl -X POST ${CLUSTER_URI}:reportFaultyNodes -d '{"nodeActions": [{"nodeId": "vtctest001-a4-0"}]}'
Enter fullscreen mode Exit fullscreen mode

(output example)

{
  "name": "projects/600975808889/locations/europe-west4/operations/6918319164184592384",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.ReportFaultyNodesOperationMetadata",
    "genericMetadata": {
      "createTime": "2025-12-07T17:02:51.863166Z",
      "updateTime": "2025-12-07T17:02:51.863166Z"
    },
    "progressMessage": "Faulty node actions received for cluster vtctest001."
  }
}
$
Enter fullscreen mode Exit fullscreen mode
$ gcloud compute instances list
Enter fullscreen mode Exit fullscreen mode

(output example)

NAME: vtctest001-login-001
ZONE: europe-west4-b
MACHINE_TYPE: n2-standard-8
PREEMPTIBLE: 
INTERNAL_IP: 192.168.0.8
EXTERNAL_IP: 34.34.124.41
STATUS: RUNNING

NAME: workstations-73e0f942-c2d3-42ee-8878-7436b5fc7e47
ZONE: asia-northeast3-b
MACHINE_TYPE: e2-standard-4
PREEMPTIBLE: 
INTERNAL_IP: 10.10.0.13
EXTERNAL_IP: 34.50.39.55
STATUS: RUNNING
$
Enter fullscreen mode Exit fullscreen mode

The faulty node is deleted.

image

Notice the GPU VM disappeared.

image

Health is also changed to “Unknown health status”.

image

To get the result of report faulty node action:

gcurl https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/operations/OPERATION_ID
Enter fullscreen mode Exit fullscreen mode

Flex-mode

gcloud compute instances create vtc-test-flex-vm \
    --project=vertex-tkim \
    --zone=europe-west4-b \
    --machine-type=a4-highgpu-8g \
    --accelerator="type=nvidia-b200,count=8" \
    --maintenance-policy=TERMINATE \
    --provisioning-model=FLEX_START \
    --max-run-duration=1d \
    --instance-termination-action=STOP \
    --image-family=common-cu124 \
    --image-project=deeplearning-platform-release \
    --boot-disk-size=200GB \
    --boot-disk-type=hyperdisk-balanced
Enter fullscreen mode Exit fullscreen mode

(output example)

image

image

image

image

Copy Name res-dws-calendar-20251208–1000kst

image

Top comments (0)