Third party REST API(OAuth) call using Azure Data Factory-Web Activity

One of the common requirements in Data flow pipelines is to retrieve data from REST endpoint and copy it to a data store.

When I started working on ADF, I found out, there was a confusing list of concepts available to implement scenarios like this:

1. Linked Service using Anonymous/Basic authentication to connect to REST endpoint.
2. Linked Service using MSI/AAD service principal to connect to REST endpoint.
3. Web Activity to fetch time based OAuth(access) token using credentials in Http POST body.

First option, helps in cases when you are trying to access Azure or third party REST APIs where no authorization(access token) is required. In this case, you can create a Linked Service using Anonymous or basic authentication to access the data available at the endpoint.

Second option lets you access only Azure APIs/services/endpoints by providing either the managed service identity or using the Service principal to authenticate and authorize using AAD.

Third option is used when you want to access a third party REST API which requires authentication as well as authorization(OAuth).In this case the Linked Service approach does not work and a web activity in a pipeline is required to fetch the access token. This access token is then used in subsequent calls to the REST endpoint.

Below are the steps to implement the third option.(Assuming Username and password are stored in Key Vault)

1. Create a Web activity to fetch username from AKV.
2. Create another Web activity to fetch password from AKV.
3. Create another Web activity to do a POST call with JSON in request body consisting of username/password. Output of this third activity gives the access token , which can be used in subsequent calls to REST API.

Refer the ADF lab 3 below , for a sample of this.

Mmodarre / AzureDataFactoryHOL

Azure Data Factory Hands On Lab - Step by Step - A Comprehensive Azure Data Factory and Mapping Data Flow step by step tutorial

Note: This is a work in progress and any feedback and collaboration is really appreciated. New excercises will be added soon.

ELT with Azure Data Factory

And

Mapping Data Flows

Hands-on lab step-by-step

Feb 2020

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any…

View on GitHub