DEV Community

Cover image for Google Cloud's Z4M launch makes storage maintenance part of the AI job plan
Da
Da

Posted on

Google Cloud's Z4M launch makes storage maintenance part of the AI job plan

Google Cloud announced general availability of its storage-optimized Z4M series on November 6, offering both virtual machines and bare-metal instances in selected regions. The launch post positions the machines for AI checkpoints, distributed key-value caches and parallel file systems, with up to 168,000 GiB of Local SSD and up to 400 Gbps networking.

For an infrastructure team, the operational detail worth examining is maintenance. Google's machine-family documentation says live migration is supported only with 42,000 GiB or less of attached Titanium SSD. It also says enabling Cloud RDMA prevents live migration: the instance must use a termination policy for host maintenance, with local SSD data persistence.

My reading is that the storage service's maintenance plan should be evaluated alongside its performance. Retained data and uninterrupted application service are different acceptance criteria.

Choose the machine and maintenance policy together

Before choosing a configuration, ask how a host maintenance event reaches the application.

Record the intended machine type, attached storage, networking configuration and host maintenance policy in the same design document. Ask the storage software supplier to confirm how its supported deployment handles that combination.

That makes the review specific. A statement about live migration for one configuration should not become an assumption about every machine in the cluster.

This is particularly useful when comparing a smaller deployment with a larger storage node. Evaluate the resulting service behavior as well as the capacity and cost.

Test whether clients can continue useful work

My recommendation is to define a maintenance acceptance exercise before production use.

Run a representative workload, initiate a supported maintenance simulation and record what the storage clients observe. Measure request failures, waiting time and recovery to the agreed service target. Track the storage software's own health state throughout the exercise.

For a checkpointing workload, include a write in progress. Determine whether it completes, retries or leaves an incomplete checkpoint that the application must reject.

For an inference cache, observe whether clients can continue through another path, recompute missing context or wait for the affected service. Those are test scenarios, not promises about Z4M or any particular storage product.

Define what makes a checkpoint usable

A fast write result alone is a limited description of checkpoint readiness.

Have the application and storage teams agree on the evidence that marks a checkpoint complete. Identify where that status is recorded and what validates the checkpoint before a resumed job uses it.

Then test the restart path. Select a completed checkpoint, resume the workload and verify that it produces the expected result. Record the elapsed time from the interruption to useful processing.

The exercise should use the same data placement and protection policy proposed for production. That lets the team distinguish a storage throughput measurement from evidence that a job can actually recover.

Keep planned maintenance and host failure separate

Google's documentation states that local Titanium SSD data is preserved during maintenance. For a host event, it describes attempts to recover attached SSD data and configurable recovery timeouts.

That distinction deserves an explicit operating decision. Ask the application owner how long a workload can wait, who decides to use another recovery path and what happens to work submitted during the interruption.

Do not infer general durability from the planned-maintenance behavior. Review the applicable storage guarantees and define the protection required by the application.

A maintenance rehearsal and a failure-recovery rehearsal answer different questions. Keep separate results for each, including any assumptions that could not be tested.

Measure performance under the intended service conditions

The launch includes vendor performance claims. Treat them as a reason to test the platform with the actual workload.

Use a representative data set, concurrency level and read/write mix. Record application completion time alongside storage and network measurements. Repeat with the protection and availability features enabled that the production design requires.

Also measure the period after a node returns. Determine when the service can accept normal work and whether background recovery changes performance.

For an AI infrastructure buyer, a useful Z4M evaluation should end with both a performance result and an operating procedure: what happens during maintenance, how completed data is recognized and how useful compute resumes. That evidence gives the team a basis for choosing a configuration it can operate.

Top comments (0)