Ayaka Hara

Posted on Apr 19, 2023

Deploying Minimum Viable Dataspace to Azure: A Step-by-Step Guide

#azure #edc

What is Minimum Viable Dataspace (MVD)?

The Minimum Viable Dataspace (MVD) is a sample implementation of a dataspace that leverages the Eclipse Dataspace Components (EDC). The main purpose is to demonstrate the capabilities of the EDC, make dataspace concepts tangible based on a specific implementation, and to serve as a starting point to implement a custom dataspace.

Since I will not explain what EDC is in this article, I recommend that those who want to learn about EDC first refer to the official Eclipse Foundation documents.

Azure deployment for MVD

In the past, there was an Azure deployment workflow on the MVD repository. However, it does not exist on the repo anymore.

There were some reasons for removing the deployment to Azure. Please refer to this GitHub issue if you want to know more details.

Some of you may still want to deploy MVD to Azure like me. In this article, how to achieve Azure deployment with the previous workflow will be illustrated.

Prerequisites
Initializing an Azure environment for CD
Create a new dataspace instance
Ready to use MVD!
Conclusion

Prerequisites

Before proceeding to the actual preparation of the environment, we must first revert to the old commit where the Deploy pipeline for Azure existed.

A fork of the MVD repository
Back to a previous commit

Please run the following commands on your local to reset the code in your fork to a previous commit where the Deploy pipeline exists. The commit id should be 9362f36f09123d7ceca67b8239cae2b82bacca3a.

git log
git reset --hard (COMMIT-ID)
git push REMOTE-NAME LOCAL-BRANCH-NAME:REMOTE-BRANCH-NAME

If a previous commit is successfully backed you will find that there are 6 yaml files (cd.yaml, check.yaml, cloud-cd.yaml, deploy.yaml, destroy.yaml, and initialize.yaml) in the workflow folder.

Prepare an Azure subscription

A GitHub workflow then needs to be run to provision the Azure resources used for CD.

Initializing an Azure environment for CD

1. Create a service identity for GitHub Actions

Further documentation for the following steps can be found in the Microsoft Docs to Configure an app to trust a GitHub repo.

1-1. Create App Registration for GitHub Actions

Name: Provide a display name for the application, e.g. "MVD GitHub Actions App"
In Supported Account Types, select Accounts in this organizational directory only.
Don't enter anything for Redirect URI (optional).

Select Register to create the app registration.

Take note of the Application (client) ID (will be required to configure a GitHub secret below).

Next, we create two types of credentials: two federated credentials to authenticate GitHub Actions, and a client secret for Terraform (required as Terraform does not yet support Azure CLI login with a service principal).
Follow the instructions to Configure a federated identity credential for the main branch and Pull requests.

1-2. Configure main Branch Credentials

Select the previously created application (e.g. "MVD GitHub Actions App") and navigate to Certificates & secrets.

Select Federated credentials, click Add credential and define the following values.

For Federated credential scenario, select GitHub actions deploying Azure resources.
For Organization, add your GitHub organization. For example, your organization is YourGitHubOrg if the URL of your GitHub repository is https://github.com/YourGitHubOrg/MinimumViableDataspace.
For Entity Type, select Branch.
For GitHub branch name, enter main.
For Name, enter a name for the credential, e.g. "mvd-main-branch".

Click Add to create the credential.

Note: You can add additional credentials to deploy from other branches.

1-3. Configure Pull Request Credentials

Now, we set up a federated credential for pull requests, which allows to run a cloud deployment to validate pull requests.

Note: This step is only required if you plan to create pull requests for the MVD.

Click Add credential to add another credential (for the previously created application (e.g. "MVD GitHub Actions App"), under Certificates & secrets, Federated credentials).

For Federated credential scenario, select GitHub actions deploying Azure resources.
For Organization, add your GitHub organization. For example, your organization is YourGitHubOrg if the URL of your GitHub repository is https://github.com/YourGitHubOrg/MinimumViableDataspace.
For Entity Type, select Pull Request.
For Name, enter a name for the credential, e.g. "mvd-pull-requests".

Click Add to create the credential.

1-4. Create a new application secret

Create a client secret by following the section "Create a new application secret" in the page on Creating a an Azure AD application to access resources.

Take note of the client secret and keep it safe.

1-5. Grant Permissions for Azure Subscription

To allow GitHub Actions to deploy resources to your Azure subscription, grant the application created above Owner permissions on your Azure subscription.
Further documentation for the following steps can be found under Assign Azure roles using the Azure portal.

Navigate to your subscription you want to deploy the MVD resources to, select Access control (IAM) and click on Add role assignment.

On the Add role assignment page, select Owner and click Next.

Now click on Select members, search for the application created above (e.g. "MVD GitHub Actions App"), click Select and then click on Next and then on Review + assign to assign the application Owner permissions on your Azure subscription.

You need to enter the full application name when searching for the application, the application will not show up if you only enter a partial name (e.g. "MVD GitHub Act" in the example above).

1-6. Configure GitHub Secrets for GitHub Actions

Finally, the application (client) ID needs to be made available to your GitHub repository using GitHub secrets.

To configure GitHub Secrets, navigate to your MinimumViableDataspace repository, select Settings, navigate to Secrets and then Actions, and click New repository secret to create a new secret.

Configure the following GitHub secrets with the value from the steps above:

Secret name	Value
`ARM_CLIENT_ID`	The application (client) ID of the application created above (e.g. "MVD GitHub Actions App").
`ARM_CLIENT_SECRET`	The application client secret.

2. Create Service Identity for MVD Runtimes

Further documentation for the following steps can be found in the Microsoft Docs to Create and configure an Azure AD application for the application runtimes.

2-1. Create App Registration for MVD Runtimes

Name: Provide a display name for the application, e.g. "MVD Runtimes App"
In Supported Account Types, select Accounts in this organizational directory only.
Don't enter anything for Redirect URI (optional).

Select Register to create the app registration.

Take note of the Application (client) ID (will be required to configure a GitHub secret below).

2-2. Create Client Secret for MVD Runtimes

Navigate to Certificates & secrets and then to the Client secrets tab (for the previously created application (e.g. "MVD Runtimes App"), and select New client secret. Create a new client secret by entering a Description (e.g. "mvd-runtimes-app-client-secret") and clicking Add.

Take note of the client secret (Value) and keep it safe (will be required to configure a GitHub secret below).

2-3. Get Application Object ID

Navigate to Azure Active Directory and select Enterprise Applications.

Take note of the enterprise application Object ID.

Make sure you use the Object ID of the Enterprise application, and not the Object ID of the App Registration!

2-4. Configure GitHub Secrets for MVD Runtimes

Configure the following GitHub secrets with the values from the steps above:

Secret name	Value
`APP_CLIENT_ID`	The application (client) ID.
`APP_CLIENT_SECRET`	The application client secret.
`APP_OBJECT_ID`	The ID of the service principal object associated with this application.

See instructions under Configure GitHub Secrets for GitHub Actions on how to configure GitHub secrets.

3. Configure CD Settings

Configure the following GitHub secrets which are required by the CD pipeline:

Secret name	Value
`ARM_TENANT_ID`	The Azure Active Directory Tenant ID. Navigate to Azure Active Directory and copy the Tenant ID from the Overview page.
`ARM_SUBSCRIPTION_ID`	The Azure Subscription ID to deploy resources to. Navigate to Subscriptions and copy the Subscription ID of your subscription.
`COMMON_RESOURCE_GROUP`	The Azure resource group name to deploy common resources to, such as Azure Container Registry. Choose any valid resource group name, e.g. rg-mvd-common.
`COMMON_RESOURCE_GROUP_LOCATION`	The location where common resources should be deployed to, e.g. eastus.
`ACR_NAME`	The name of the Azure Container Registry to deploy. Use only lowercase letters and numbers.
`TERRAFORM_STATE_STORAGE_ACCOUNT`	The name of the storage account used to store the Terraform state container, e.g. mvdterraformstates.
`TERRAFORM_STATE_CONTAINER`	The name of the container used to store the Terraform state blob, e.g. mvdterraformstates.

4. Deploying CD resources

Common resources need to be deployed once, these resources will be used by all CD pipelines.

Manually run the Initialize CD GitHub Actions workflow and make sure that it passes successfully.

Since workflow_dispatch is already in deploy pipeline a ‘Run workflow’ button should be on the Actions tab, enabling you to easily trigger a run. (GitHub Actions: Manual triggers with workflow_dispatch
) If you cannot find the button like the screenshot below I would suggest you to try changing the file extension, from deploy.yaml to deploy.yml.

Your infrastructure is now set up to run deployments, you can now run the Deploy GitHub Actions workflow in the next step.

Create a new dataspace instance

Once your environment is set up, follow these steps to create a new dataspace instance:

Select the Deploy GitHub Actions workflow and provide your own resources name prefix. Please, use at most 3 characters, composed of lower case letters and numbers.

Click on Run workflow to manually run the Deploy GitHub Actions workflow.

Make sure that it passes successfully.

Ready to use MVD!

Once Deploy pipeline passes successfully then you can now access to MVD which just has been deployed with your own resources name prefix.

The Data Dashboard is a web application (development UI) on top of EDC's DataManagementAPI and is deployed for each participant. It can be accessed at the URLs provided in the GitHub workflow run page like below.
For instance, if the name prefix was dragon:

You will have three Data Dashboards for three participants: company 1, company 2 and company 3 as below.

company 1

company 2

company 3

If you want to demonstrate the possibilities of Dataspaces please refer to Vision Demonstrator document.

Conclusion

Azure deployment workflow has been removed from the MVD repo but it still can be deployed with the previous commit. Please be aware of that it will not be updated anymore.
Hope this step-by-step guide is helpful for some of you who want to try MVD on Azure.

DEV Community