loading...

Use Container for Azure Batch Service

kenakamu profile image Kenichiro Nakamura ・4 min read

Azure Batch service is one of great choice to run your program in parallel.

In this article, I will explain how to use container as your batch program.

Why container?

The service is nothing special but uses bunch of VMs as compute pool and run your program in parallel. This means that these VM should have all required components to run your application.

You can register "applications" to share the components between VMs, but it still require some pre-work.

If you have too much dependencies or version conflict between VM OS components and your program, container is the best solution to simplify the issue, as container includes everything you need and deployed as a whole.

Azure Batch with Container

If you are a programmer or prefer coding to GUI, refer to official doc: container workloads where you can find detail explanation with sample python/C# code.

I will use Azure Portal (GUI) instead as some IT folks prefer GUI, including me.

Azure resources

I provisioned following resources for the scenario.

  • Azure Batch Account
  • Azure Storage Account
  • Azure Container Registry (ACR)

Alt Text

Obtain ACR key

I need credential information for ACR when configuring Azure Batch pool.

1. Go to ACR | Access keys and enable "Admin user".

Alt Text

2. Copy Login server, username and password for following steps.

Create pool

Next, create Azure Batch computer pool with container support.

1. Go to Azure Batch service account | Pools and click "Add".

Alt Text

2. Start creating pool. You can use different image as long as it supports container. See here for more detail.

I configured with following settings.

  • Pool ID: You can set whatever you want
  • Image Type: Market place
  • Publisher: microsoft-azure-batch
  • Offer: ubuntu-server-container
  • SKU: 16-04-lts

3. At "Container configuration, select Custom and click "Container registries".

Alt Text

4. Click "Add" and enter registry information obtained above and click "OK".

Alt Text

5. Select added registry and click "Select".

6. I keep "Container image names" blank for now as I don't have any image yet.

7. Select VM size and Scale to fit your needs. I selected the smallest VM with 2 low-priority nodes.

Alt Text

Register image

Let's add a sample program and push to ACR. You can create any docker image as you want. As I am C# developer, I created C# console app which simply print "Hello" with first argument.

Program.cs

using System;

namespace samplebatch
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine($"Hello {args[0]}");
        }
    }
}

Dockerfile

FROM mcr.microsoft.com/dotnet/core/runtime:3.1-buster-slim AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["samplebatch/samplebatch.csproj", "samplebatch/"]
RUN dotnet restore "samplebatch/samplebatch.csproj"
COPY . .
WORKDIR "/src/samplebatch"
RUN dotnet build "samplebatch.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "samplebatch.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
#ENTRYPOINT ["dotnet", "samplebatch.dll"]

As I don't need the container to auto run the program, I commented out ENTRYPOINT.

I run the container locally and it worked as expected.

Alt Text

Upload the image to ACR. I tagged it "v1".

Alt Text

Create job and task

Finally, create Azure Batch job and task(s).

1. Go back to Azure Batch account | Jobs and click "Add".

Alt Text

2. Create job.

  • Job ID: any name you want
  • Pool: select created pool which supports container

Alt Text

3. Select created job, then click "Add" in Tasks.

Alt Text

4. Give any name for Task ID.

5. Enter "dotnet samplebatch.dll ken" to command line. This command will be executed inside the docker container.

6. Enter full name for image name. In my case, it is "azurebatchwithcontainer.azurecr.io/samplebatch:v1".

7. Add "--rm --workdir /app" to "Container run options".

8. Click Submit.

Confirm the result

Wait until the task state becomes "Completed", then I can confirm the result.

1. Make sure the state becomes "Completed".

Alt Text

2. Open the task and open "stdout.txt".

Alt Text

3. Confirm the result.

Alt Text

What's important

There are several places I stuck when I tried this.

  • image name needs to be full name including server name and tag.
  • command line is executed inside the docker image.
  • if working directory is not root, use --workdir option.
  • When update code, use different tag, otherwise the pool won't download the new one as it reuses cache if name is same.

You can still use "ENTRYPOINT" and just specify argument if you want, so just play with it to figure out.

Additional information

In addition to what I explained here, you can use other Azure Batch features.

  • Resource files
  • Environment Settings
  • Saved output files, etc.

The main difference is that you need to use container support pool and specify **Container settings" when creating task.

Summary

If you struggle with pre-task and environment setup for Azure Batch, try container to simplify it!

Discussion

pic
Editor guide