Building a Vertex AI custom job container

As part of the DeepCell benchmarking project, we need to accelerate and especially automate generating predictions on data.

On Vertex AI, we think that means using Custom Jobs–which can be triggered via API–and custom containers. (docs)

It was relatively simple to create & upload the container. Here's the code (permalink).

The main thing that surprised me was that it took 27min end-to-end. About 20min of that was downloading & especially extracting the base Google TensorFlow 2.8 image. 😤 (it was several gigabytes)

If this pans out, we'll have an easy way to issue benchmarking runs programmatically. That'll be a real time saver, meaning we'll be able to gather more data.

Here's the container uploaded to the artifact registry 😎

Extra bonus points if the Experiments framework works out…