DEV Community

Kenichiro Nakamura
Kenichiro Nakamura

Posted on • Updated on

Use Container for Azure Batch Service

Azure Batch service is one of great choice to run your program in parallel.

In this article, I will explain how to use container as your batch program.

Why container?

The service is nothing special but uses bunch of VMs as compute pool and run your program in parallel. This means that these VM should have all required components to run your application.

You can register "applications" to share the components between VMs, but it still require some pre-work.

If you have too much dependencies or version conflict between VM OS components and your program, container is the best solution to simplify the issue, as container includes everything you need and deployed as a whole.

Azure Batch with Container

If you are a programmer or prefer coding to GUI, refer to official doc: container workloads where you can find detail explanation with sample python/C# code.

I will use Azure Portal (GUI) instead as some IT folks prefer GUI, including me.

Azure resources

I provisioned following resources for the scenario.

  • Azure Batch Account
  • Azure Storage Account
  • Azure Container Registry (ACR)

Alt Text

Obtain ACR key

I need credential information for ACR when configuring Azure Batch pool.

1. Go to ACR | Access keys and enable "Admin user".

Alt Text

2. Copy Login server, username and password for following steps.

Create pool

Next, create Azure Batch computer pool with container support.

1. Go to Azure Batch service account | Pools and click "Add".

Alt Text

2. Start creating pool. You can use different image as long as it supports container. See here for more detail.

I configured with following settings.

  • Pool ID: You can set whatever you want
  • Image Type: Market place
  • Publisher: microsoft-azure-batch
  • Offer: ubuntu-server-container
  • SKU: 16-04-lts

3. At "Container configuration, select Custom and click "Container registries".

Alt Text

4. Click "Add" and enter registry information obtained above and click "OK".

Alt Text

5. Select added registry and click "Select".

6. I keep "Container image names" blank for now as I don't have any image yet.

7. Select VM size and Scale to fit your needs. I selected the smallest VM with 2 low-priority nodes.

Alt Text

Register image

Let's add a sample program and push to ACR. You can create any docker image as you want. As I am C# developer, I created C# console app which simply print "Hello" with first argument.

Program.cs

using System;

namespace samplebatch
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine($"Hello {args[0]}");
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Dockerfile

FROM mcr.microsoft.com/dotnet/core/runtime:3.1-buster-slim AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["samplebatch/samplebatch.csproj", "samplebatch/"]
RUN dotnet restore "samplebatch/samplebatch.csproj"
COPY . .
WORKDIR "/src/samplebatch"
RUN dotnet build "samplebatch.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "samplebatch.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
#ENTRYPOINT ["dotnet", "samplebatch.dll"]
Enter fullscreen mode Exit fullscreen mode

As I don't need the container to auto run the program, I commented out ENTRYPOINT.

I run the container locally and it worked as expected.

Alt Text

Upload the image to ACR. I tagged it "v1".

Alt Text

Create job and task

Finally, create Azure Batch job and task(s).

1. Go back to Azure Batch account | Jobs and click "Add".

Alt Text

2. Create job.

  • Job ID: any name you want
  • Pool: select created pool which supports container

Alt Text

3. Select created job, then click "Add" in Tasks.

Alt Text

4. Give any name for Task ID.

5. Enter "dotnet samplebatch.dll ken" to command line. This command will be executed inside the docker container.

6. Enter full name for image name. In my case, it is "azurebatchwithcontainer.azurecr.io/samplebatch:v1".

7. Add "--rm --workdir /app" to "Container run options".

8. Click Submit.

Confirm the result

Wait until the task state becomes "Completed", then I can confirm the result.

1. Make sure the state becomes "Completed".

Alt Text

2. Open the task and open "stdout.txt".

Alt Text

3. Confirm the result.

Alt Text

What's important

There are several places I stuck when I tried this.

  • image name needs to be full name including server name and tag.
  • command line is executed inside the docker image.
  • if working directory is not root, use --workdir option.
  • When update code, use different tag, otherwise the pool won't download the new one as it reuses cache if name is same.

You can still use "ENTRYPOINT" and just specify argument if you want, so just play with it to figure out.

Additional information

In addition to what I explained here, you can use other Azure Batch features.

  • Resource files
  • Environment Settings
  • Saved output files, etc.

The main difference is that you need to use container support pool and specify **Container settings" when creating task.

Access Resource files

When I assign resources files into a task, these files are stored in Node, not in the container. But when Azure Batch service instantiate the container instance, it maps all folder in the node to the container.

You can find environment variables by following tutorial here and the wd folder path is stored in AZ_BATCH_TASK_WORKING_DIR environment variable.

For example, I can read all text files from the directory.

var batch_working_dir = Environment.GetEnvironmentVariable("AZ_BATCH_TASK_WORKING_DIR");
// Read text file
foreach (string file in Directory.EnumerateFiles(batch_working_dir, "*.txt", SearchOption.AllDirectories))
{
    string contents = File.ReadAllText(file);
    Console.WriteLine(contents);
}
Enter fullscreen mode Exit fullscreen mode

Same apply for other files and outputs.

Summary

If you struggle with pre-task and environment setup for Azure Batch, try container to simplify it!

Discussion (4)

Collapse
karthik1036 profile image
karthik1036 • Edited

I have .net core Console Application which prints a basic hello world message I have published my application to azure container registry
my Docker File
FROM mcr.microsoft.com/dotnet/core/runtime:3.1-buster-slim AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["ConsoleApp1.csproj", ""]
RUN dotnet restore "./ConsoleApp1.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "ConsoleApp1.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "ConsoleApp1.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "ConsoleApp1.dll"]\

I have created a node please go through this image for better understanding node-configuration1 ,node-configuration,node-config,task-config(please see in the stack overflow url)

I am using a Linux ((ubuntu container)container for integrating with azure batch but whenever I run My task its saying to install
It was not possible to find any installed .NET Core SDKs Did you mean to run .NET Core SDK commands? Install a .NET Core SDK from: aka.ms/dotnet-download

Task_issue

how to install dotnet core in linux container please guide me on this through sharing some
resource

please see the images in the stack overflow:stackoverflow.com/questions/658841...

Collapse
kenakamu profile image
Kenichiro Nakamura Author

Sorry to get back to you late. First thing first, if you docker build the Dockerfile in your local computer and run the docker image, does it all work fine?

Collapse
karthik1036 profile image
karthik1036

how to run container image in azure batch (Task) when I run this command in docker
docker run -it demoapplication1234.azurecr.io:443/demoapplication

it navigates to different directory c:/app and i am passing arguments and getting the result

*how to pass arguments in azure batch and and how should i pass docker run command in azure batch task command line *

Collapse
kenakamu profile image
Kenichiro Nakamura Author

You can set "dotnet [args]" in command line.
Then set demoapplication1234.azurecr.io:443/demoapplication: to image name and add container option as '--rm --workdir c:/app'

You cannot do -it as batch is not for interactive operation.