DEV Community

Cover image for Azure Data Factory Automation — Adding Log Analytics for Monitoring
Shashank Banerjea
Shashank Banerjea

Posted on

Azure Data Factory Automation — Adding Log Analytics for Monitoring

One of the challenges, that I often faced with automating deployment of Azure Data Factory has been to add monitoring to the Azure Data Factory in a scriptable way.

The monitoring is essential to broad system health. For example, to generate alerts failures or if a pipeline is running longer than expected. The preferred and documented way of generating this alert has been to push the logs to Azure Log Analytics and analyzing the logs to generate alerts or provide a broad monitoring capability integrated with other parts of the infrastructure on Azure.

The Azure Data Factory documentation does provide a method using the Azure Monitor REST API. However it is little unwieldy to use, , especially in CI/CD pipelines.

There is another method to do this is actually that is well documented but not often referred to. It is available in the documentation for Azure Monitor.

So, associating an Log Analytics as a diagnostics log and metrics sink for Azure Data Factory using Azure CLI becomes as simple as running the script below:

az monitor diagnostic-settings create \
 - name LogAnalytics02-Diagnostics \
 - resource /subscriptions/(your-subscription)/resourceGroups/(your-resource-group)/providers/Microsoft.DataFactory/factories/(data-factory-name) \
 - logs '[{"category": "PipelineRuns","enabled": true}]' \
 - metrics '[{"category": "AllMetrics","enabled": true}]' \
 - workspace /subscriptions/(your-subscription)/resourcegroups/(your-resource-group)/providers/microsoft.operationalinsights/workspaces/(your-log-analytics-workspace-name)
Enter fullscreen mode Exit fullscreen mode

The pre-requisite to run this scripts are - Azure Data Factory Instance and Log Analytics workspace should already be provisioned.

Just substitute the values for subscriptions and resources with values that applies to you. The example above shows just adding the logs for PipelineRuns but you can add more values in JSON string for logs parameter.

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read full post →

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay