DEV Community: Anirudh Garg

Why use Azure Functions for ML inference ?

Anirudh Garg — Thu, 28 May 2020 16:57:18 +0000

This article highlights the current limitations in offerings by FaaS providers for ML inference applications and how capabilities in Azure Functions overcome these making it an ideal platform for such applications. It also links to an end to end sample illustrating these features.

ML Inference in FaaS Services

The core promise of Serverless/FaaS (Functions as a Service) computing is to make it easier for developers to build, deploy and scale up their applications. Among the different kinds of applications - ML applications have seen an explosion in their usage and are now rapidly being deployed and bringing unprecedented novel capabilities in various domains.

A typical ML workflow consists of using sample data to train a model and then use that model in an application to make predictions based on input. These stages are typically called training and inference respectively. On the surface, ML inference should be a straightforward FaaS use case as essentially there is a single "predict" function that is called to "infer" the result for a particular input. However, due to various limits that typical FaaS platforms impose it hasn't been as optimum a platform that it can be. These are some of the limitations:

1. **ML libraries can be big**

ML(especially Deep learning) libraries can be big in size. For example the PyTorch (even the non-CUDA version) library is around ~300MB.
However, most FaaS providers only allow a maximum of ~250MB-~500MB of deployed package size including any modules needed in the application. Customers have to resort to getting around to these limitations by compressing shared libraries, removing test folders etc.

2. **Python ML libraries use native dependencies**

Many Python ML libraries call into other C/C++ libraries, those need to be recompiled based on the OS causing a mismatch between the development and production deployments. For instance, developers using Windows have to resort to using Docker before deploying their FaaS application.

3. **ML models can be big**

Some of the latest ML models can be very big in size - for example the AlexNet pre-trained PyTorch model is 230MB. Some models like GPT-2 can be several GB in size. FaaS offerings have a limited deployment size package as mentioned above - further, they don't offer a way to separate out the model from the code by mounting a file system for instance.

Further, some large models need to be completely loaded into memory before they can be used and amount of memory offered in current Serverless offerings might not be enough.

4. Concurrency models for I/O bound invocations are sub-optimal

Many ML inference API's are typically I/O bound, however, in current FaaS execution frameworks a single instance cannot handle concurrent requests. The can lead to a lot of wasted idle resources on individual instances.

5. Cold start

FaaS workloads scale down to zero instances when not being used, therefore starting a new instance always incurs lag - which is called cold start. For many applications this lag is not acceptable.

6. GPU support is not available

Many inference applications have a very stringent latency requirements (think for instance applications related to providing auto-suggestions as you type). To meet these latency requirements often (especially for bigger models) GPU processing can lead to several times of improvement. However FaaS providers currently do not support GPU's.

7. No native support for long running ML workloads and/or orchestration patterns

Some ML workloads might be inherently long running and/or have complex orchestration needs. Current FaaS providers do not offer native frameworks for these scenarios and customers need to resort to external services and built the complex coordination logic themselves.

Azure Functions Python

In Azure Functions Python capabilities and features have been added to overcome some of the above limitations and make it a first class option for ML inference with all the traditional FaaS benefits of scaling on demand, fast iteration and pay-for-use.

1. Higher deployment package sizes

Azure Functions support higher deployment packages as much as several GB in size. This means that even larger deep learning frameworks like TensorFlow and PyTorch can be supported out of the box without resorting to having to reduce the size without affecting functionality.

2. Support for Remote Build

As mentioned above, many Python libraries have native dependencies. This means if you are developing your application on Windows and then deploy it in production which typically is on Linux - it might not work as they have to be recompiled to work on another operating system.

To help with this problem in Azure Functions we built the Remote Build functionality, where we automatically compile and pull in the right libraries on the server side. Hence, they don't have to be included in your deployment package locally and only need to be referenced in your requirements.txt file. This is default experience when using VSCode or the func core tools to deploy your Python Azure Functions application. This is a snippet from the deployment logs showing the libraries being installed on the server side.

12:02:54 AM pytorch-image-consumption: Starting deployment...
.
12:03:01 AM pytorch-image-consumption: Running pip install...
...
12:03:14 AM pytorch-image-consumption: [07:03:14+0000] Collecting torch==1.4.0+cpu
12:03:14 AM pytorch-image-consumption: [07:03:14+0000]   Downloading https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl (127.2MB)
12:03:45 AM pytorch-image-consumption: [07:03:45+0000] Collecting torchvision==0.5.0
12:03:45 AM pytorch-image-consumption: [07:03:45+0000]   Downloading https://files.pythonhosted.org/packages/1c/32/cb0e4c43cd717da50258887b088471568990b5a749784c465a8a1962e021/torchvision-0.5.0-cp37-cp37m-manylinux1_x86_64.whl (4.0MB)
....
12:04:08 AM pytorch-image-consumption: [07:04:08+0000] Successfully installed azure-functions-1.2.1 certifi-2020.4.5.1 chardet-3.0.4 idna-2.9 numpy-1.15.4 pillow-7.1.2 requests-2.23.0 six-1.14.0 torch-1.4.0+cpu torchvision-0.5.0 urllib3-1.25.9

3. Support for mounting an external file system through Azure Files

To allow for arbitrarily large ML models and to separate the workflow of building the application and training your ML model the capability to automatically mount a configured Azure file share has been recently introduced.

This is as simple as using an Azure CLI command where the $shareName Azure File share will automatically be mounted whenever the $functionAppName Function app starts up. The path available to the Function app will be with the name of $mountPath.

az webapp config storage-account add \
  --resource-group myResourceGroup \
  --name $functionAppName \
  --custom-id $shareId \
  --storage-type AzureFiles \
  --share-name $shareName \
  --account-name $AZURE_STORAGE_ACCOUNT \
  --mount-path $mountPath \
  --access-key $AZURE_STORAGE_KEY

This is an Azure Files storage view with two different models.

This functionality is available both in the Consumption and Premium plans. It is also worth noting that Azure Files have a Premium tier which can be used as well - it offers a higher level of performance i.e. high throughput and low latency for IO-intensive workloads(many ML applications)

The above two features Remote Build and support for Azure Files allow the deployment package to be much smaller. For example, in the PyTorch sample referenced below the total package size would have been close to 520MB in size when using the resnet100 model (~350MB for PyTorch and ~170MB for the model) while without it it is barely 50KB in size.

4. Azure Functions concurrency model is optimal for I/O bound invocations

In Azure Functions - the same instance is re-used for concurrent calls. This can lead to optimum use of resources especially for concurrent I/O bound invocations where the underlying api has async variants.

5. Azure Functions Premium SKU's offer higher memory limits and no cold start

The Azure Functions Premium SKU offers higher memory limits - up to 14GB for cases where you would want to load the entire model in memory.

SKU	Cores	Memory	Storage
EP1	1	3.5GB	250GB
EP2	2	7GB	250GB
EP3	4	14GB	250GB

It also avoids cold start for those applications where there cannot be any latency at all.

Further, the custom container support enables integration with the Azure Machine Learning service such that you can package a model from the Azure ML service into an Azure Function Premium app and allows the use of a Conda environment.

6. Deploy Azure Functions to a Kubernetes cluster with GPU support

Finally, as in Azure Functions the programming model is separate from the deployment target, an Azure Function app can be deployed to a Kubernetes cluster (it can be scaled up and down using KEDA). Many managed Kubernetes offering (including AKS) have GPU support.

7. Orchestrating inference invocations - Durable Functions Python

With the impending release of Durable Functions Python long running ML workloads can now be supported. Durable Functions allows many different orchestration use cases including processing multiple inputs in a batch, parallelizing classification tasks etc. These are enabled by the Chaining, Fan-Out Fan-In and other Durable Function patterns. This sample illustrates the use of the Fan-Out Fan-In pattern doing parallel image classification using TensorFlow.

Sample

To put this all together and show a real example - please find a step by step walk through examples of an Azure Function app that is able to use several of these pre-trained PyTorch models from an Azure File share to create an inference application which classifies images.

It illustrates the use of Remote Build, gives guidance on how you can deploy the same app to the Premium plan for no-cold start and higher memory limits and gives guidance on how the same app can be deployed to a AKS cluster with GPU. More end to end examples will be added soon for text generation and other ML tasks.

Hopefully this blog post encourages you to consider Azure Functions for your ML inference applications. Please ask any questions in the comments, or open any issues in the Azure Functions Python Worker repo or in the sample repo.

Announcing Azure Functions Kafka extension Public Preview

Anirudh Garg — Tue, 28 Apr 2020 04:38:18 +0000

We're excited to announce today the public preview of the Kafka Extension for Azure Functions. With this new extension you can now have functions trigger in response to messages in Kafka Topics, or write to a Kafka Topic through the output binding. The extension is supported when hosting functions in the Premium plan enabling it to elastically scale and trigger on Kafka messages. You can also use Kafka with Azure Functions containers in Kubernetes alongside KEDA

How to get started with the Kafka Trigger

To get started with using the Kafka trigger, you need to include the extension in your function project.

.NET Functions

For .NET functions, you can pull in the Kafka NuGet extension.

dotnet add package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

JavaScript, TypeScript, Python, Java, and PowerShell Functions

For other functions, you need to install the extension into your project using the Azure Functions core tools

func extensions install Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

You can then create a function that can activate and run whenever a message is dropped in a Kafka topic.

Example C# Trigger using Confluent Cloud Managed Kafka

public static class kafka_example
    {
        [FunctionName("kafkaApp")]
        public static void ConfluentCloudStringTrigger(
             [KafkaTrigger(
                "BootstrapServer",
                "users",
                ConsumerGroup = "<ConsumerGroup>",
                Protocol = BrokerProtocol.SaslSsl,
                AuthenticationMode = BrokerAuthenticationMode.Plain,
                Username = "<APIKey>",
                Password = "<APISecret>",
                SslCaLocation = "confluent_cloud_cacert.pem")]
        KafkaEventData<string> kafkaEvent,
        ILogger logger)
        {       
            logger.LogInformation(kafkaEvent.Value.ToString());
        }
    }

Example host.json

{
  "version": "2.0",
  "extensions": {
    "kafka": {
      "maxBatchSize": 100
    }
  }
}

You can find more settings documented here

Example functions.json

Note: This is auto-generated for C#

{

  "bindings": [
    {
      "type": "kafkaTrigger",
      "consumerGroup": "azfunc",
      "protocol": "saslSsl",
      "authenticationMode": "plain",
      "username": "<KafkaAPIKey>",
      "password": "<KakfaSecret>",
      "sslCaLocation": "<RootCertificateFile>",
      "topic": "<KafkaTopicName>",
      "brokerList": "<Server>.eastus.azure.confluent.cloud:9092",
      "name": "kafkaEvent"
    }
  ],

}

Please find a complete end to end walkthrough and sample app using the Confluent Cloud in this GitHub repo Kafka extension sample with Confluent Cloud.

This extension is being developed in the open, please contribute, try out and post any issues on the Azure Functions Kafka extension GitHub repo.

Along with the Rabbit MQ Trigger that we announced last year that is now also supported in the Premium Plan, we continue to innovate to bring in more Cloud Native and Open Source event sources. We are looking forward to see what applications you build with this !

Scale a HTTP Triggered app up and down in Kubernetes using KEDA and Prometheus

Anirudh Garg — Mon, 13 Jan 2020 03:22:19 +0000

This blog post goes into how you can scale up and down your Http Triggered application in Kubernetes based on requests using KEDA and Prometheus.

The application here is an Azure Function app but it can be any app that exposes a Http interface. Where appropriate below I point out where you can use your app instead of an Azure Function app.

Among the major cloud providers, the FaaS implementation in Azure, Azure Functions, is unique that its runtime is open-source. The runtime and your code can therefore be deployed to a custom container or deployed on your own infrastructure including Kubernetes.

To enable scaling of a function app (or any other workload) in Kubernetes we at Azure (along with RedHat) built KEDA, Kubernetes Event Driven Autoscaling. With the combination of aforementioned runtime and KEDA you can run and scale your Azure Functions in your own Kubernetes cluster. Currently in KEDA we support more than twenty different message event sources including Kafka, RabbitMQ, NATS, Azure Queue, AWS SQS Queue, GCP Pub Sub etc. However, there is no support for Http request based scaling. This post outlines one approach on how you can scale a Http Trigerred function app in Kubernetes using the Prometheus KEDA scaled object and an Ingress Controller.

Overview

The basic idea is that we will deploy an Ingress Controller in this case the NGINX Ingress Controller and have all HTTP traffic to your function app go through it. We use Prometheus to track the incoming request metrics on the Ingress Controller. Finally, we use KEDA's Prometheus based scaler to scale up/down the function app deployment.

Walkthrough

Prerequisites

A Kubernetes cluster which has the ability to install a Service with a Load Balancer (usually any cloud provider). The steps below were tested using an AKS cluster.
kubectl pointing to your Kubernetes cluster
Helm to install the artifacts. All of the artifacts below use Helm3
Azure Functions core tools
docker installed locally and a Docker Hub account.

Steps

Create a namespace for your ingress resources
```
kubectl create namespace ingress-nginx
```

Install the NGINX-Ingress ingress controller

Use Helm to deploy an NGINX ingress controller and enable metrics and set the right annotations for Prometheus. The ingress controller is installed as service with LoadBalancer type. In addition, a backend Service and a metrics Service are also deployed.

helm install ingress-controller stable/nginx-ingress \     
    --namespace ingress-nginx \
    --set controller.replicaCount=2 \
    --set controller.metrics.enabled=true \
    --set controller.podAnnotations."prometheus\.io/scrape"="true" \
    --set controller.podAnnotations."prometheus\.io/port"="10254"

kubectl -n ingress-nginx get svc
NAME                                                  TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
ingress-controller-nginx-ingress-controller           LoadBalancer   10.0.14.166    40.70.230.xxx   80:31036/TCP,443:32179/TCP   31s
ingress-controller-nginx-ingress-controller-metrics   ClusterIP      10.0.240.199   <none>          9913/TCP                     31s
ingress-controller-nginx-ingress-default-backend      ClusterIP      10.0.63.133    <none>          80/TCP                       31s

The ingress controller is exposed via the EXTERNAL-IP 40.70.230.xxx above. Also have a look at the following page for instructions on how to install it for various configurations.

Optionally - Create a DNS entry pointing to your ingress controller. For AKS, you can get a cloudapp.azure.com address using the procedure here. In the steps below the fqdn configured is "function-helloworld.eastus2.cloudapp.azure.com"

Deploy Prometheus to monitor the NGINX ingress Controller

kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/    

kubectl -n ingress-nginx get svc
NAME                                                  TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
...
prometheus-server                                     NodePort       10.0.38.50     <none>          9090:30860/TCP               34s

kubectl -n ingress-nginx get pods
NAME                                                              READY   STATUS    RESTARTS   AGE
..
prometheus-server-86cd54f9d5-9xxh7                                1/1     Running   0          95s

Note: If you happen to use any other namespace other than "ingress-nginx" then you need to go and change the namespace in the yaml files from here and then deploy.

Deploy KEDA

helm install keda kedacore/keda --namespace ingress-nginx     

kubectl get pods -n ingress-nginx
NAME                                                              READY   STATUS    RESTARTS   AGE
...
keda-operator-697b98dcdd-8zdrk                                    2/2     Running   0          5d17h

Deploy a function app to the Kubernetes cluster
Create a Python Http Trigerred function app, generate the required docker file and finally deploy the function app to the cluster.
```
func init --worker-runtime python
func new --template "HttpTrigger" --name helloworld
func init --docker-only
func kubernetes deploy --name function-helloworld --namespace ingress-nginx --service-type ClusterIP --registry anirudhgarg
```
Note that the authentication mode has to be changed to anonymous for now while we are working to support function keys. Navigate to the function app folder and open the function.json file. Find the following line: "authLevel": "function" and change the authLevel to "anonymous"

--name is the name of your Deployment. --registry points to your DockerHub registry and you have to be logged in to docker and connected to your account locally. See more here

Note: Instead of using a function app you can deploy your own app that listens to Http requests. Just make sure you create a k8s Cluster IP Service pointing to your deployment.

Deploy an Ingress Resource pointing to the deployed function app Service.

This is how the YAML looks like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: function-helloworld
  namespace: ingress-nginx
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2    
spec:  
  rules:  
    - host: <replace-with-host-name-pointing-to-ingress-controller>
      http:
        paths:                
        - backend:
            serviceName: function-helloworld-http
            servicePort: 80
          path: /helloworld(/|$)(.*)

You can find an example here

The serviceName attribute is the name of the Service for the function app. host should point to the fqdn configured pointing to the Ingress Controller. You can also choose a random name here but a host has to be configured otherwise Prometheus monitoring of the Ingress Resource will not work. A path has also been configured with prefix "helloworld".

Create an Ingress Resource with NGINX Ingress Controller annotations pointing to the Prometheus Service

This is how the YAML looks like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus-service
  namespace: ingress-nginx
  annotations:
    kubernetes.io/ingress.class: nginx    
spec:
  rules:  
    - http:
        paths:
        - backend:
            serviceName: prometheus-server
            servicePort: 9090
          path: /

The serviceName attribute is the name of the Service for the Promtheus Server.

kubectl apply -f "https://raw.githubusercontent.com/anirudhgarg/azurefunction-k8s-http/master/prom-ingress.yml"

Deploy the KEDA Prometheus Scaled object which inturn monitors the NGINX Ingress Controller

kubectl apply -f "https://raw.githubusercontent.com/anirudhgarg/azurefunction-k8s-http/master/keda-prom.yml"

This is how the YAML looks like:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
name: prometheus-scaledobject
namespace: ingress-nginx
labels:
    deploymentName: function-helloworld-http
spec:
scaleTargetRef:
    deploymentName: function-helloworld-http
pollingInterval: 15
cooldownPeriod:  30
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: prometheus
  metadata:
    serverAddress: http://prometheus-server.ingress-nginx.svc.cluster.local:9090
    metricName: access_frequency
    threshold: '1'
    query: sum(rate(nginx_ingress_controller_requests[1m]))

deploymentName is the name of the function app Deployment, the pollingInterval is how frequently in seconds does KEDA poll Prometheus, we have a minimum of 1 pod (minReplicaCount) and the maximum scale out is 10 pods (maxReplicaCount). query is pointing to the prometheus query which tracks the metrics to incoming requests to the ingress controller in the last minute. Since the threshold is '1' the function app will scale as long as the number of requests/minute > 60.

Test !
The function app is now listening on:
http://function-helloworld.eastus2.cloudapp.azure.com/helloworld/api/helloworld?name=anirudh

Note that if you did not create a domain name pointing to your Ingress Controller then you might need to use curl --resolve or its equivalent to invoke the function app

```
 curl -v function-helloworld.eastus2.cloudapp.azure.com/helloworld/api/helloworld?name=anirudh
* Trying 40.70.230.199...
* TCP_NODELAY set
* Connected to function-helloworld.eastus2.cloudapp.azure.com (40.70.230.199) port 80 (#0)
> GET /helloworld/api/helloworld?name=anirudh HTTP/1.1
> Host: function-helloworld.eastus2.cloudapp.azure.com
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.2
< Date: Mon, 13 Jan 2020 01:47:30 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 14
< Connection: keep-alive
<
Hello anirudh!* Connection #0 to host function-helloworld.eastus2.cloudapp.azure.com left intact
```

Now you can use your favorite Http requests tool and send requests at a high enough rate to triger the scale out. (I used a tool called k6)

```
kubectl -n ingress-nginx get pods
NAME                                                              READY   STATUS              RESTARTS   AGE
function-helloworld-http-6ccd9c9bbf-6f6d7                         0/1     ContainerCreating   0          0s
function-helloworld-http-6ccd9c9bbf-98bzq                         1/1     Running             0          15s
function-helloworld-http-6ccd9c9bbf-dcdwc                         0/1     ContainerCreating   0          0s
function-helloworld-http-6ccd9c9bbf-fr7hq                         0/1     ContainerCreating   0          0s
function-helloworld-http-6ccd9c9bbf-k9lhn                         1/1     Running             0          6d20h
function-helloworld-http-6ccd9c9bbf-mfp4c                         1/1     Running             0          15s
function-helloworld-http-6ccd9c9bbf-v7g47                         0/1     ContainerCreating   0          0s
function-helloworld-http-6ccd9c9bbf-x9l2t                         1/1     Running             0          15s
ingress-controller-nginx-ingress-controller-6c9f7486d4-27vjq      1/1     Running             0          7d1h
ingress-controller-nginx-ingress-controller-6c9f7486d4-b8ddr      1/1     Running             0          7d1h
ingress-controller-nginx-ingress-default-backend-df57464f-lvqmj   1/1     Running             0          7d1h
keda-operator-697b98dcdd-8zdrk                                    2/2     Running             0          6d18h
prometheus-server-86cd54f9d5-9xxh7                                1/1     Running             0          7d1h
```

After a while if there are no further requests the function pods will scale back down to 1. Note that we are only scaling down to 1 here.

Hope you found this useful. Please try it out and let me know in the comments or send me a tweet if this approach worked for you or not. We are looking to streamline this process further as we go forward.

Acknowledgments for inspiration - Autoscaling Kubernetes apps with Prometheus and KEDA post by Abhishek Gupta, and to OpenFaaS which also uses Prometheus metrics for request based scaling.