Building Managed Lustre Storage on Google Cloud (2/3)

Setting Up Managed Lustre Storage on Google Cloud (2/3)

Table of Contents

1. Overview
2. Excerpts from Official Documentation
3. Hands-on: Command Execution and Troubleshooting

Previous: Setting Up Managed Lustre Storage on Google Cloud (1/3)

2. Excerpts from Official Documentation

Google Cloud > Vertex AI > Storage

The Vertex AI Training Cluster (VTC) service integrates with two high-performance storage solutions:

• Filestore: A managed file service that provides a shared /home directory across all nodes in the cluster.
• Managed Lustre: A parallel file system engineered for extreme I/O performance, making it ideal for training runs on massive datasets.

On VTC, Filestore is required, while Managed Lustre is optional. For large clusters, Managed Lustre is strongly recommended because its high I/O throughput boosts overall cluster performance. Compared to the cost of the AI accelerators (such as GPUs) powering these HPC clusters, high-performance storage is relatively inexpensive. Choosing high-performance storage can therefore lower the overall Total Cost of Ownership (TCO) for large-scale training jobs.

Google Cloud > Vertex AI > Storage > Storage Integration for Training Clusters

Filestore is required to provide the cluster's shared /home directory, whereas Managed Lustre is an optional high-performance file system for throughput-intensive workloads. Correctly configuring the network for these storage services prior to deploying the cluster is critical.

Filestore for Home Directories

The service uses a Filestore instance to serve the cluster's shared /home directory. To ensure proper connectivity, create your cloud resources in the following order:

1. Create a VPC network: First, deploy a VPC network configured with the recommended MTU (for example, 8896).
2. Create a Filestore instance: Next, deploy a Filestore instance inside the VPC you just created.
3. Create a training cluster: Finally, deploy the cluster so it can connect to the Filestore instance within the same network.

Google Cloud Managed Lustre for High-Performance Workloads

Google Cloud Managed Lustre is Google Cloud's fully managed, high-performance parallel file system service.

It is specifically designed to accelerate heavy AI/machine learning and high-performance computing (HPC) workloads.

For workloads requiring peak I/O performance, you can attach a Managed Lustre file system.

The service connects to your VPC using Private Service Access.

Critical Networking Limitation: No Transitive Peering

Both Filestore and Google Cloud Managed Lustre share an important limitation: neither supports transitive peering.

This means only resources inside the VPC directly connected to the storage service can access it. For example:

• Even if the cluster's VPC (N1) is peered with the storage service, another VPC (N2) peered with N1 cannot access that storage.

Storage Integration for Training Clusters

Configuring the network for your selected storage services before deploying the cluster is essential.

(To stay focused on Lustre, Filestore-specific details from the official documentation are omitted here. Refer directly to the linked documentation if needed.)

Lustre

Managed Lustre offers four performance tiers, each providing a different maximum throughput per TiB. The performance tier also dictates the minimum and maximum instance sizes, as well as the allowed capacity increments. You cannot change an instance's performance tier after creation.

Deploying Managed Lustre requires Private Service Access. This feature establishes VPC peering between the training cluster's VPC and the VPC hosting Managed Lustre using a dedicated /20 subnet.

When creating a Managed Lustre instance, define the following properties:

• The instance name used in Google Cloud.
• The file system name used by client-side tools (for example, lfs).
• Storage capacity in gibibytes (GiB). Capacity ranges from 9,000 GiB to approximately 8 PiB (7,632,000 GiB). The maximum instance size depends on the performance tier.
• Managed Lustre provides performance tiers ranging from 125 MBps to 1,000 MBps per TiB.
• For optimal performance, create the instance in the same zone as the training cluster.
• The VPC network used by this instance must be the same network used by the training cluster.

Configure a Managed Lustre Instance (Optional)

To achieve peak performance on a training cluster, deploy Google Cloud Managed Lustre in the same VPC and zone as the training cluster, and configure VPC peering through Service Networking.

Create a Lustre Instance

gcloud lustre instances create LUSTRE_INSTANCE_ID \
  --project=PROJECT_ID \
  --location=ZONE \
  --filesystem=lustrefs \
  --per-unit-storage-throughput=500 \
  --capacity-gib=36000 \
  --network=NETWORK_NAME

Google Cloud > Vertex AI > Storage > Storage Integration for Training Clusters > Lustre

Google Cloud Managed Lustre provides a fully managed, high-performance parallel file system optimized for AI and HPC applications. With multi-petabyte capacities and up to 1 TBps of throughput, Managed Lustre makes it easy to migrate high-performance workloads to the cloud.

Managed Lustre instances reside in a zone within a region. A region is a specific geographic location where resources run, and each region is divided into multiple zones. For example, the us-central1 region in the central United States contains zones us-central1-a, us-central1-b, us-central1-c, and us-central1-f. For more information, see Geography and regions.

To reduce network latency, creating Managed Lustre instances in a region and zone close to your workloads is recommended.

For VTC, achieving optimal performance requires creating all cluster components in the exact same region and zone—not just nearby. That is, "deploy Managed Lustre in the same VPC and zone as the training cluster, and use VPC peering via Service Networking."

Next: Setting Up Managed Lustre Storage on Google Cloud (3/3)