End-to-end Correlation Across Logic Apps

In this blog post, I would like to discuss achieving end-to-end correlation across Logic Apps, a need to have a solution while dealing with an Enterprise application involving multiple Logic Apps. Let us understand the concept with the help of a real-time scenario.

The Scenario- Employee On-boarding System
Here we consider an organisation that has a well-defined Employee Onboarding process. Multiple Logic Apps are involved in building this Onboarding application. Below is the illustration of the Logic Apps engaged in the application.

The HR portal compiles the Employee details into a Service Bus message which gets processed by the back end, automated using Logic Apps. The ‘Onboard’ is the one that defines the core workflow. This Logic App, in turn, calls out to various other Logic Apps to achieve actions like Background Verification (BGV), Employee Addition to Organisation Databases (Addition), Enrolling Employee for Insurance (Insure). Based on the success of the underlying process, the ‘Onboard’ Logic App will either complete the Service Bus message or dead letter the same. The Logic App definition is as below:

What is Correlation?
Correlation is the mechanism for relating workflows to each other. For example, Background verification followed by adding that Employee detail to the Organisation database in the Onboarding workflow as above. Commonly, each of the workflow mentioned above can be a Logic App. Hence business workflow definitions would involve multiple Logic Apps. In which case, end to end correlation across these Logic Apps becomes critical.

Why Is It Needed?
When the application workflow spans multiple Logic Apps, there is a need for the end-to-end correlation to get the much-needed business visibility.

For example, in the scenario mentioned above, the message containing Employee details flows through 4 Logic Apps to complete the onboarding process. It would be essential for the business to understand if all the four stages of the process get through or not.

It is essential to understand where the message got stuck in the end-to-end execution if we simplify the requirement.

How Is Correlation Implemented in Logic Apps?
Logic App implicitly can correlate the runs using a Correlation Id. The ‘Onboard’ Logic App invokes the other 3 Logic Apps, BGV, Addition and Insure, in the scenario above. When we investigate their run histories, we can observe a unique correlation id associated with every new transaction.

Below is the details section of a specific run of the Logic App ‘Onboard’. Observe the highlighted Correlation Id in the image below.

Please Note: Above is the screenshot from Serverless360 that highlights the run details of a Logic App in a Business Application. In the Azure portal, find the same as below:

Observe the same Correlation ID in the Logic App, BGV:

The Correlation ID remains the same in the corresponding run details of the other two Logic Apps too:

This way, every Employee processing is associated with a unique Correlation ID by default.

Hence it is possible to correlate runs across multiple Logic Apps using the Correlation ID and recognise them as a complete transaction.

Auto-generated vs Custom Correlation Ids
When this Correlation ID is good to have, the autogenerated GUID is not that useful. It is hard to interpret the actual Employee that is getting onboarded. What can be helpful is, configuring the Employee ID itself as the Correlation ID. And this is possible!!!

We can custom define the Correlation ID by setting the parameter “x-ms-client-tracking-id”. In this case, I am choosing to put the Employee ID to this parameter. If the Logic App is HTTP triggered, then this value can be set in the header. In my case, it is a Service Bus Queue triggered Logic App; hence I had defined this as a user property on the Service Bus message.

Upon setting this property, I observe the correlation ID is set as the Employee ID, just as expected. PFB the screen capture that displays the correlation ID set as the Employee ID, which is more beneficial.

This correlation Id flows through the Logic Apps BGV, Addition and Insure as well:

Correlating Messages over Logic Apps
I intend to correlate all the transactions corresponding to an employee by the Employee ID. Let us first explore one solution available in the Azure portal itself.

The Logic Apps Management Solution allows you to monitor and manage your Logic Apps. This solution provides you with a summary of the overall health of your Logic Apps, with options to drill into details and troubleshoot unexpected behaviour patterns.

With rich, out of the box views, you can get insights into crucial processing, including:

• Summary of all logic app runs.
• Ability to drill into details of the logic app runs by their names or by their processing status.
• Logic app run details including their status, execution times and unique identifiers to establish correlation across Logic Apps.
• Ability to identify resubmitted runs of a logic app.

Pre-requisite: To take advantage of this solution, Logic Apps should enable Log Analytics to push diagnostics data to the OMS workspace.

With the Employee ID set as the Correlation ID, I could filter the records by Employee ID in the Tracking ID column. Below is the list of actions carried out for Employee with ID, 133595.

I am happy to attain this correlated result with the minimal configuration of custom defining the Correlation ID on the Service Bus message that kick starts the ‘Onboard’ Logic App. Indeed impressive!!!

Serverless360 BAM to Correlate Logic Apps
Logic App Management’s above result is simple to set up and does provide the much-needed correlation. However, this could be complex for the HR team to handle, and it might not be practical to share them with access to the Azure portal. Considering the HR team as the business users in this context, how can we securely provide them with the end-to-end visibility on the message flowing through this Serverless Application? How can we help them figure out what happened with onboarding a specific Employee?

There is a need for the organisation to build an end-to-end traceability/monitoring solution that business users can use in any integration solution. However, tracking is often an afterthought, and the operational support team struggles to run the implemented solution. Sometimes the delivery team will work on a solution. Still, often it is bespoke and challenging to use across projects, and teams spend about 20-30% of their time addressing this challenge, building custom solutions like logging components and web dashboards. The solution will be premature most of the time since creating enterprise-grade tracking/monitoring solutions requires enormous effort.

That is precisely the challenge we are tackling in Serverless360 with the BAM feature.

Let us see how the Employee Onboarding process can be well defined and tracked in Serverless360 BAM

Model your business processes, transactions, stages and map them to physical processing elements in integration solutions.

Define properties to track business values at run-time. Correlate the flow of data within your system. Configure reprocess to restore failures.

This implementation requires instrumentation in Azure Logic Apps, which Serverless360 facilitates with its Custom Logic App Connectors:

• Start Transaction: This stage marks the beginning of the instrumentation. In our scenario, this occurs in the ‘Onboard’ Logic App when the Service Bus message is received.

• Checkpoint with Correlation: When the subsequent stage occurs in another Logic App, use this connector to correlate using the Employee ID as below

When the subsequent stage appears in the same Logic App, use the checkpoint connector. In our case, the last stage is in the ‘Onboard’ logic App that marks the process completion can either be successful or cause a business exception based on the onboarding status. By configuring the property SL360-Exception it is possible to capture the business exceptions as below.

Upon implementation of Serverless360 BAM, the business can enjoy the following benefits:

• Search for specific transaction based on a tracked property (Employee ID in this case)
• Focused view of all failed transactions in the ‘Action Required’ tab
• Assign a transaction to a team member to act on the same!
• Monitoring on saved queries and for exceptions
• Reprocess the failures to restore business.
• Dashboard full of Insights from the tracked info

Serverless360 BAM is not limited to Logic Apps; you can Perform end-to-end tracking on business process flow across Azure resources and hybrid integrations.

Conclusion
To summarise, we discussed correlation and its need considering a real-time scenario of the Employee On-Boarding Process. We explored the default correlation id in the Logic Apps and discovered how to make it usable by custom defining the same. When I assigned the Employee ID to the Correlation ID, it did solve our requirement of correlating the process across multiple Logic Apps using the Logic App Management. Technical users can manage and monitor the Logic Apps in the Azure portal using Logic App Management. However, for business users to get end-to-end visibility, better insights, proactive monitoring, and reprocessing toolset to restore business, Serverless360 BAM turns out to be the best solution.