Enabling Pipelines: Easier than ever

Ashwin Ganesh Kumar — Wed, 20 Mar 2024 12:01:55 +0000

Application Lifecycle Management (ALM)strategy is a topic that you keep encountering more if you are building applications/automations in Power Platform.

There are multiple ways to implement ALM for power platform. But I want to focus on deploying solutions using pipelines. So far, you must be an admin to set up and manage a pipeline. This process requires the admin to install the power platform pipelines and setup pipelines for the end user to implement them.

But now we can create personal pipelines using the platform host. At the time of writing this piece, it is still a preview feature, so please don't do this in your production environment. This feature stimulates you to skip the manual export/import process, and that is a good step in enabling citizen developers to implement ALM practices.

How to implement it?

For my test use case, I have two environments, one being Dev, and the other being Test. I have a solution that contains a demo app.

When you select the export solution button, I see a card about the deployment process (preview)..

When you are doing it for the first time it takes a few seconds for the setup to be completed. Once the setup is complete, you see a screen which you might already be familiar with (if you have used power platform pipelines before).

Now we are ready to create a new pipeline for our use case. Here you can select your target environment, for now you can only select one target environment. You can filter the target environments in terms of Your role, Data platform, and Environment type.

After creating a pipeline, the next step is to then deploy it to our target environment.

We can also schedule our deployment if needed. After this step it first validates the solution and gives us a summary in which you can add some deployment notes for future references.

Once the deployment is succesfull, you can find your solution(managed) in the target environment. You can also manage your own pipelines, which is a model driven app. It contains essential information like the setup, run history, artifacts etc.

But this preview feature does come with some limitations (at least for now):

We cannot extend the personal pipelines.
We can select only one target environment.
Pipelines remain personal to you and can't be shared.

Conclusion

Personal pipelines do enable citizen developers to move away from the manual export/export process. Even though it's a preview feature, it has enormous potential.

Bulk Deletion: Sharepoint Items

Ashwin Ganesh Kumar — Wed, 10 Jan 2024 11:52:12 +0000

Recently, I encountered a daunting challenge: clearing thousands of records from an existing Sharepoint list. The typical UI deletion method was impractical for such a massive task, sparking my quest for a more efficient solution. In this blog, I'll share the three methods I experimented with in my test environment, offering an insight into the pros and cons of each. Whether you're facing a similar hurdle or just looking to enhance your Sharepoint management skills, these strategies might pave the way for smoother, more effective list handling. Let's explore these solutions together.

Power Automate: Delete Item

This method is very simple and staright forward.

This method should work just fine, but it's important to be aware of its limitations and potential impacts:

Limited Retrieval Capacity: The Get Items action can only get 5000 items at a time. So then you might want to run the flow multiple times or place the Get Items in a loop to get more items.
API Call Consumption: As stated in the documentation, this will consume a lot of API calls and might hamper your other processes.
Speed: It deletes items individually, and the flow takes a long time to run.

Power Automate: Sharepoint Rest API

This method is a bit complex and involves a lot of steps. We are going to make use of Sharepoint Rest API. With the help of google I was able to find an informative blog on batch calls. There you will find loads of information about CRUD operations using $Batch

From the docs, we can get code snippet of how a batch DELETE request could look like:

POST https://fabrikam.sharepoint.com/_api/$batch HTTP/1.1
Authorization: Bearer <access token omitted>
Content-Type: multipart/mixed; boundary=batch_7ba8d60b-efce-4a2f-b719-60c27cc0e70e
Host: fabrikam.sharepoint.com
Content-Length: 647
Expect: 100-continue

--batch_7ba8d60b-efce-4a2f-b719-60c27cc0e70e
Content-Type: multipart/mixed; boundary=changeset_efb6b37c-a5cd-45cb-8f5f-4d648006e65d

--changeset_efb6b37c-a5cd-45cb-8f5f-4d648006e65d
Content-Type: application/http
Content-Transfer-Encoding: binary

DELETE https://fabrikam.sharepoint.com/_api/Web/lists/getbytitle('OldList') HTTP/1.1
If-Match: "1"

--changeset_efb6b37c-a5cd-45cb-8f5f-4d648006e65d--
--batch_7ba8d60b-efce-4a2f-b719-60c27cc0e70e
Content-Type: application/http
Content-Transfer-Encoding: binary

GET https://fabrikam.sharepoint.com/_api/Web/lists HTTP/1.1

--batch_7ba8d60b-efce-4a2f-b719-60c27cc0e70e--

Now let's try to replicate this in our flow.

I first want to store the values of the site address and listname in a variable. And I also intialize varItemCount which we will use later in the flow to loop through the lists and perform the delete action till the list is empty.

Now we want to split the code snippet into chunks to formulate our request to the sharepoint API. Firstly, let's make a changset request.

After which we will use the Get items action to retrieve the top 1000 items from our list. Now we will replace the Id in the changeset request to the Id from the Get items action.

Now we could use the Send an HTTP request to Sharepoint to perform our batch DELETE operation using the POST method

I tested this method with a sharepoint list which had around 2350 records and the time it took finish the operation was around 3 minutes.

Pros

Efficiency: We can handle larger batches of deletions more effeciently than individual item deletion.
Control: We have more control and flexibility.

Cons

Complexity: Requires the understanding of Sharepoint REST API and JSON. It costs me some digging to understand the logic.

Power Automate: Graph API

I explored this method more out of curiosity. Recently I came across multiple GraphAPI contents, and I wanted to see if I could also do some POST operations with Graph. The simple answer is YES.

To perform an operation using Graph we need to understand how to work with Graph API. Microsoft documentation was a good starting point and also this blog from Lewis.

To get your own hands with Graph, go to Graph Explorer. Here we are able to play with sample data provided by Microsoft or authenticate with our tenant(provided we have rights) to run queries.

In our case, we need to have permission to access the tenant, the sharepoint site and the list, and also perform delete operation via the Graph API.

Register an application with azure AD:
First we create a new app registration from Azure portal. In the
API permissions tab, we add delegated permissions for the Microsoft Graph API(Sites.ReadWrite.All).
Flow Steps
- Now we create an instant flow with manual trigger.
- Initialize Variable for the token URL. (https://login.microsoftonline.com/{TenantId}/oauth2/v2.0/token)
- HTTP action to receive the token. For the body part of the HTTP action. client_id=your_app_client_id&scope=https%3A%2F%2Fgraph.microsoft.com%2F.default&client_secret=your_app_secret&grant_type=client_credentials A secure way to use the app credentials would be to store them in a key vault and use the Azure key vault action in the flow to retrieve them.
- After getting the token we add another HTTP action to retrieve the items from the list that we want to delete. For the URL (getting the first 500 items) https://graph.microsoft.com/v1.0/sites/{site-id}/lists/{list-id}/items?top=500 For the headers we need to fill the key as Authorization and the value as Bearer {your_access_token}. You can retrieve the token from the output of the previous step.
- We now add Parse JSON action to parse the outputs from our previous step to get all the retrieved items.
- Now we want to run through every individual item to do the delete operation, for which we create an apply to each action and pass the id output from the Parse Json step.
- Within the apply to each we add a HTTP action to perform the DELETE method to our retrieved items from the list. URL: https://graph.microsoft.com/v1.0/sites/{site-id}/lists/{list-id}/items/@{items('Apply_to_each')?['id']}

Even though this method works we still have to keep certain things in mind:

Premium: We are using the HTTP connector which is a premium connector in Power Platform.
Graph API access: We need to get access to our tenant to perform queries and operations with Graph API.
App Registration: We need appropriate rights in azure to create app registrations and setting the right API permissions for Graph.
Rate limits: Graph API has rate limiting, that might slow down or interrupt deletion for large batches.

In this method I haven't tried batching using graph API, but this is something I would like to explore.

Conclusion

In conclusion, managing large-scale deletions in Sharepoint can be a complex task, but with the right tools and knowledge, it can be simplified and made more efficient. In this blog, we explored three distinct power automate methods: Power Automate Delete Item, Sharepoint REST API, and Power Automate Graph API. Each method has its strengths and limitations.

The Power Automate Delete Item is straightforward and user-friendly but is limited by its retrieval capacity and speed. The Sharepoint REST API offers more efficiency and control, especially for larger batches, but requires a deeper understanding of Sharepoint's inner workings. Lastly, the Power Automate Graph API provides a modern approach, albeit with considerations such as premium connector usage and API rate limits.

Your choice of method will depend on your specific needs, the size of the data, and your comfort level with Sharepoint and API interactions. Remember, efficient Sharepoint management is not just about knowing what tools are available but also understanding the nuances of each to make an informed decision.

Feel free to experiment with these methods in your environment and see which works best for your situation.

DEV Community: Ashwin Ganesh Kumar

Enabling Pipelines: Easier than ever

How to implement it?

Conclusion

Bulk Deletion: Sharepoint Items

Power Automate: Delete Item

Power Automate: Sharepoint Rest API

Power Automate: Graph API

Conclusion