Automate document processing with Form Recognizer and Logic Apps (Part 2)

Processing of forms and documents is part of several scenarios both in business and in everyday life. Manual data extraction from documents, either in electronic or printed format, is time-consuming, costly, and error-prone.

Azure Form Recognizer is an Applied AI Service that enables you to extract text, table data, key-value pairs, and layout information from forms and documents. In this two-part series, you will learn how to build an end-to-end document processing automation solution utilizing Azure Form Recognizer, Logic Apps, Azure Cosmos DB, and Power BI.

In the first part, you trained two custom models for extracting key-value pairs from customer service review forms and composed these models together into a single model. In this article, you will build an automated form processing solution.

You will learn how to:

• Create a Logic App workflow that responds to Event Grid events.
• Integrate Form Recognizer into a Logic Apps workflow.
• Store the extracted information from the Form Recognizer’s model in Azure Cosmos DB.
• Use Power BI to visualize the insights from the analysis of the forms.

To build this solution, you will need an Azure subscription. If you don’t have one, you can sign up for an Azure free account.

Scenario details

Consider that you work in a company that provides customer service to a variety of customers. Every day, customers leave reviews about their experiences with the customer service they received. You need to analyze these reviews in order to identify areas of improvement and to track customer satisfaction. These reviews are submitted as paper forms or PDFs and may include up to two different form types. To make this process easier, we need to use a single service to analyze all the reviews.

In the following image, you can see the workflow that we will build to automate the process of extracting and analyzing customer service reviews. In this post, we will focus on the second and third steps of the workflow.

Architecture

The following architecture diagram illustrates the main components involved in the automated form processing solution that we are building and the information flow. The system receives the forms (either in PDF or scanned image format), extracts the fields from the form and saves the extracted data in Azure Cosmos DB. Power BI is then used to visualize the insights from the data.

Dataflow

The information flow corresponding to the above architecture diagram is described as follows:

1. PDF forms or scanned images are (manually or programmatically) uploaded to a container in Azure Storage Account.
2. Whenever a form is uploaded to the specified container, an Event Grid event will trigger the Logic App to start processing the form.
3. The Logic App sends the URL of the file to the Form Recognizer API and receives the response data.
4. The extracted key-value pairs are saved into a NoSQL database in Azure Cosmos DB.
5. Power BI is connected to Azure Cosmos DB to ingest the extracted data and provide dashboards.

Components

Setup Azure Blob Storage

You will create a storage account and a container to upload the forms that will be processed by our Logic App.

1. Sign in to the Azure portal, search for Storage accounts and then select Create.

2. Create a storage account with the following settings:

   • Subscription: Your Azure subscription.
   • Resource group: Select an existing resource group (i.e. the resource group that you created in the Part 1) or create a new one.
   • Storage account name: Enter a unique name.
   • Region: Choose any available region, for example, West Europe.
   • Performance: Standard.
   • Redundancy: Locally-redundant storage (LRS).

3. Select Review + Create and then select the Create button and wait for the deployment to complete. Once the deployment is complete, navigate to your storage account.

4. In the left pane, under Data storage, select Containers.

5. Create a new container. Select a name and set the public access level to Container.
   

Setup Event Grid

Event Grid is a kind of event broker that you can use to integrate applications using events. We will use Event Grid as a trigger to run the Logic App when a file is uploaded in the Azure Storage container.

To use Event Grid, you must first register it as a resource provider in your subscription.

Setup Azure Cosmos DB

You need to create an Azure Cosmos DB account, database, and container to store the fields extracted from the safety forms.

1. In the Azure portal, search for Azure Cosmos DB and then click Create.
2. Create a new Azure Cosmos DB for NoSQL account by selecting the corresponding card in the Which API best suits your workload? window.

3. Create an Azure Cosmos DB account with the settings below:

   • Subscription: Your Azure subscription.
   • Resource group: Select an existing resource group (i.e. the resource group that you created in the Part 1) or create a new one.
   • Account name: Enter a unique name.
   • Location: Choose any available region, for example, West Europe.
   • Capacity mode: Provisioned throughput.
   • Limit total account throughput: selected.

4. Select Review + Create and then select the Create button and wait for the deployment to complete. Once the deployment is complete, navigate to your resource.

5. In the left pane, select Data Explorer. Then, create a new database and a container.
   

Create a Logic App

The last component that we need to provision is a Logic App.

1. In the Azure portal, search for Logic apps and then click + Add.

2. Create a Logic App by specifying the following parameters:

   • Subscription: Your Azure subscription.
   • Resource group: Select an existing resource group (i.e. the resource group that you created in the Part 1) or create a new one.
   • Logic App name: Enter a unique name.
   • Region: Choose any available region, for example, West Europe.
   • Plan type: Consumption.
   • Zone redundancy: Disabled.

3. Select Review + Create and then Create.

4. Once the deployment is complete, navigate to your Logic App resource. You are now ready to build the document processing workflow.

Build the workflow

You will build the document processing workflow using the Logic App Designer, a graphical user interface that enables you to create workflows visually.

1. Once the Logic App resource is created, you will see the starter window of Logic App designer. In the Start with a common trigger window, select the When an Event Grid event occurs block.
   

   If you can’t find this block, create an empty workflow and then select the When a resource event occurs trigger.

2. Sign in to your account. In the When a resource event occurs block specify the following details:

   • Subscription: Select your subscription.
   • Resource Type: Microsoft.Storage.StorageAccounts
   • Resource Name: The name of your Storage account.
   • Event Type Item – 1: Microsoft.Storage.BlobCreated

   In the Event Grid trigger block, you can optionally apply filters. For example, you can use the Prefix Filter to subscribe to events from a specific container in the Storage account or use the Suffix Filter to filter events based on the extension of the uploaded file, such as .jpg or .pdf.

3. Add a new step. In the Choose an operation window, search for Parse JSON and then select the respective action.

4. In the Parse JSON block, enter the following parameters:

   • Content: Body
   • Schema: Use the following schema.

      {
          "properties": {
              "data": {
                  "properties": {
                      "api": {
                          "type": "string"
                      },
                      "blobType": {
                          "type": "string"
                      },
                      "clientRequestId": {
                          "type": "string"
                      },
                      "contentLength": {
                          "type": "integer"
                      },
                      "contentType": {
                          "type": "string"
                      },
                      "eTag": {
                          "type": "string"
                      },
                      "requestId": {
                          "type": "string"
                      },
                      "sequencer": {
                          "type": "string"
                      },
                      "storageDiagnostics": {
                          "properties": {
                              "batchId": {
                                  "type": "string"
                              }
                          },
                          "type": "object"
                      },
                      "url": {
                          "type": "string"
                      }
                  },
                  "type": "object"
              },
              "dataVersion": {
                  "type": "string"
              },
              "eventTime": {
                  "type": "string"
              },
              "eventType": {
                  "type": "string"
              },
              "id": {
                  "type": "string"
              },
              "metadataVersion": {
                  "type": "string"
              },
              "subject": {
                  "type": "string"
              },
              "topic": {
                  "type": "string"
              }
          },
          "type": "object"
      }
   

   

5. Find the Analyze Document for Prebuilt or Custom models (v3.0 API) in the Form Recognizer connector and add it to your workflow.
   

6. Connect to Form Recognizer:

   • Connection name: Enter a name for the connection.
   • Endpoint URL: Enter the Endpoint URL of your Form Recognizer resource.
   • Account Key: Enter the Key 1 of your Form Recognizer resource.

   You can find the Endpoint URL and the Key in your Form Recognizer resource, under Keys and Endpoint.

7. Then, specify the following parameters:

   • Model Identifier: Enter the name of the model that will be used to analyze the forms (i.e. the name of the composed model that you created in Part 1).
   • Document/Image URL: Click Add dynamic content and select the url.

Save the extracted data

The JSON response of the Analyze Document for Prebuilt or Custom models (v3.0 API) action includes several information, such as the text extracted from the form, layout information, selection marks, and key-value pairs. We are interested in extracting the fields and their associated values. In the following steps, you will build a pipeline that extracts the fields from the JSON response, processes the values, generates a JSON file and saves the file in Azure Cosmos DB.

1. Add a Parse JSON action and specify the following details:

   • Content: Select Add dynamic content and find the block called fields.
   • Schema: To generate the schema, use a sample JSON response of the Form Recognizer action. Click Use sample payload to generate schema and paste the section of the Form Recognizer’s response identified by the term "fields".

   The Logic App automatically adds a For each block around the Parse JSON block.

2. Add a Compose action (inside the For each block). We will save the extracted fields in the following format:
   

   {
     "BillingPayment": "value1",
     "Comments": "value2",
     "OtherService": "value3",
     "ProductServiceInformation": "value4",
     "Recommend": "value5",
     "Satisfaction": "value6",
     "Suggestions": "value7",
     "TechnicalSupport": "value8",
     "id": "id"
   }
   

3. To extract the value associated with each field, use if statements. You can add an if statement by selecting Add dynamic content > Expression.

   For example, to extract the values associated with the fields BillingPayment, Comments and Recommend, you can use the below expressions:
   

   // BillingPayment
   if(and(contains(body('Parse_Fields')?['BillingPayment'], 'content'), equals(body('Parse_Fields')?['BillingPayment']?['valueSelectionMark'], 'selected')), 1, 0)
   
   // Comments
   if(contains(body('Parse_Fields')?['Comments'], 'content'), body('Parse_Fields')?['Comments']?['content'], 'NA')
   
   // Recommend
   if(and(contains(body('Parse_Fields')?['RecommendVeryLikely'], 'content'), equals(body('Parse_Fields')?['RecommendVeryLikely']?['valueSelectionMark'], 'selected')), 'Very likely', if(and(contains(body('Parse_Fields')?['RecommendLikely'], 'content'), equals(body('Parse_Fields')?['RecommendLikely']?['valueSelectionMark'], 'selected')), 'Likely', if(and(contains(body('Parse_Fields')?['RecommendNeutral'], 'content'), equals(body('Parse_Fields')?['RecommendNeutral']?['valueSelectionMark'], 'selected')), 'Neutral', if(and(contains(body('Parse_Fields')?['RecommendUnlikely'], 'content'), equals(body('Parse_Fields')?['RecommendUnlikely']?['valueSelectionMark'], 'selected')), 'Unlikely', if(and(contains(body('Parse_Fields')?['RecommendVeryUnlikely'], 'content'), equals(body('Parse_Fields')?['RecommendVeryUnlikely']?['valueSelectionMark'], 'selected')), 'Very unlikely', 'NA')))))
   

   In a similar way, you can extract the values corresponding to the remaining fields.
   

4. Click Add an action and select the Create or update document (V3) action of the Azure Cosmos DB connector. Then, configure the Azure Cosmos DB connection by specifying the values below:

   • Connection name: Enter a name for the connection.
   • Authentication type: Access Key.
   • Access Key to your Cosmos DB account: Enter the primary (or secondary) key of your Azure Cosmos DB resource.
   • Account ID: Enter the name of your Azure Cosmos DB account.

   To find the primary key of your Cosmos DB account, navigate to your Cosmos DB resource and select Keys.

5. In the Create or update document (V3) block, specify the ID of your database and collection. Then, under Document, select Add dynamic content and add the block named Outputs.
   

Save the workflow and then click Run Trigger > Run. Upload a file to your Azure Storage container to test the Logic App.

Connect Power BI to Azure Cosmos DB

You can use Power BI to visualize the results obtained from the form processing workflow. If you don’t have a Power BI subscription, you can use Power BI Desktop, which is a free service.

1. Open Power BI Desktop and in the Home tab select Get data > More…
2. Choose the Azure Cosmos DB v1 connection and click Connect.
3. In the pop-up window, enter the URL of your Cosmos DB account and the id of your database and collection. Then, click OK.
4. Once Power BI is connected to your Cosmos DB account, you can see the stored data and transform it.

Below you can see a simple Power BI dashboard that I created to visualize the extracted data.

Summary and next steps

In this article, you created an end-to-end automated form processing solution using Form Recognizer, Logic Apps, Azure Cosmos DB and Power BI. You can use this solution to create automated workflows for your specific scenarios. You can also extend this scenario by adding AI capabilities (such as, sentiment analysis, key-phrase extraction, opinion mining) in your Logic App workflow.

Here are some additional scenarios that leverage Azure Form Recognizer:

• Use Azure Functions and Python to process stored documents – Microsoft Docs
• Create Form Recognizer workflows with AI Builder – Microsoft Docs

Clean-up

If you have finished learning, you can delete the resource group from your Azure subscription:

1. In the Azure portal, select Resource groups on the right menu and then select the resource group that you have created.
2. Click Delete resource group.

---

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineering student and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub

Comments

[John Antony]

@sfoteini Have you used connected container setup for the form recognizer?Just wanted to know if we create a custom model and used connected container where the model will get created

[Foteini Savvidou]

I have not used connected containers for Azure Form Recognizer, but I found the following information in Microsoft Docs:

• Document Intelligence connected containers send billing information to Azure by using a Document Intelligence resource on your Azure account.
• Azure AI containers are not licensed to run without being connected to Azure for metering.
• Connected containers don't send customer data, such as the image or text that's being analyzed, to Microsoft.

From my understanding, all customer data, including custom models, will stay in the local environment.