CI/CD for Automating SP Migration with ShareGate

ShareGate application allows scheduling tasks, so the migration is executed outside of office hours.
The process may also be automated using PowerShell, which is especially important if content transformation must be executed in the process.

I decided to go one step further and automate it using Azure Pipelines.

Prerequisites

ShareGate is not available as a Marketplace extention and cannot be easily used in Microsoft-hosted pipelines. The software has to be installed and activated with a license key- which is assigned per user per workstation.

In order to automate the migration process, I am using a self-hosted agent configured on a Virtual Machine with ShareGate installed and properly licensed.
The process is using the concept presented in the Schedule your SharePoint migration using PowerShell and ShareGate with the only difference being the trigger.

Pipelines

Extract and Transform

The first stage of the migration process exports content from the SharePoint lists and performs transformations.
The ShareGate_ExportToExcel.yml job is using the self-hosted agent with Sharegate installed, whilst the TransformData.yml job is executed on a Microsoft-hosted agent:

jobs:
- job: ${{parameters.jobName}}
  displayName: 'ShareGate Export to Excel'
  dependsOn: ${{ parameters.dependsOn }}
  timeoutInMinutes: 360
  pool: ShareGate # IMPORTANT
  variables:
  - group: ${{parameters.groupKeyVaultName}}
  - name: scriptsLocation
    value: "$(Build.SourcesDirectory)/Pipelines/scripts"

  steps:
  - checkout: self
    persistCredentials: true
  - task: powershell@2
    name: SGMigration
    inputs:
      filePath: $(scriptsLocation)/SG-ExportToExcel.ps1
      arguments: '-userName_SP ''$(SP-SvcAcct-Name)'' -userPassword_SP (ConvertTo-SecureString -AsPlainText $(SP-SvcAcct-Pswd) -Force) `
          -listNames ''Portfolio'', ''Schedule'', ''Archive'' '
    displayName: Export to Excel
```

#### Load to downstream environments

The next stages execute the content migration to selected downstream environments.

During test migrations, I'm reusing the previously exported and transformed Excel, even if the data may be slightly outdated. Being able to use the same Excel file is saving me a lot of time; it took 1,5 hours to generate it

The import process consists of several steps:

- updating lists configuration to, for example, allow older item versions which may be missing values in the fields that are currently mandatory
- importing content using source SharePoint site and the transformed Excel file
- resetting lists configuration to its previous state
- validating list configuration and list content

```yml
jobs:
# set the list configuration: required columns, fillIn choices etc
- template: jobs_SG_PreMigration.yml
  parameters:
    environment: ${{parameters.environment}}
    groupVariablesName: ${{parameters.groupVariablesName}}
    groupKeyVaultName: ${{parameters.groupKeyVaultName}}
    jobName: SPOPreMigration

# import with ShareGate
- template: jobs_SG_Migration.yml
  parameters:
    environment: ${{parameters.environment}}
    groupVariablesName: ${{parameters.groupVariablesName}}
    groupKeyVaultName: ${{parameters.groupKeyVaultName}}
    jobName: Import
    dependsOn: SPOPreMigration

# set the list configuration back: required columns, fillIn choices etc
- template: jobs_SG_PostMigration.yml
  parameters:
    environment: ${{parameters.environment}}
    groupVariablesName: ${{parameters.groupVariablesName}}
    groupKeyVaultName: ${{parameters.groupKeyVaultName}}
    jobName: SPOPostMigration
    dependsOn: Import

# run list validation
- template: jobs_SPO_Validate.yml
  parameters:
    environment: ${{parameters.environment}}
    groupVariablesName: ${{parameters.groupVariablesName}}
    groupKeyVaultName: ${{parameters.groupKeyVaultName}}
    jobName: SPOValidate
    searchOrphanedUsers: true
    dependsOn: SPOPostMigration
```

Once again, only the import has to be executed on the self-hosted agent

```yml
jobs:
- deployment: ${{parameters.jobName}}
  displayName: 'ShareGate Migration'
  environment: ${{ parameters.environment }}
  dependsOn: ${{ parameters.dependsOn }}
  timeoutInMinutes: 360
  pool: ShareGate # IMPORTANT
  strategy:
    runOnce:
      deploy:
        steps:
        - checkout: self
          persistCredentials: true
        # Copy results
        - task: CopyFiles@2
          condition: always()
          displayName: 'Copy SG mapping files'
          inputs:
            contents: |
              $(Build.SourcesDirectory)\SPO_Migration\ShareGateConfig\*
            targetFolder: 'D:\ShareGateConfig'
            flattenFolders: true
            overWrite: true

        - task: powershell@2
          name: SGMigration
          inputs:
            filePath: $(scriptsLocation)/SG-Migrate.ps1
            arguments: '-tenantName ''$(Az_TenantName)'' -siteName ''$(SPO_SiteName)'' `
                -userName_SPO ''$(SPO-SvcAcct-Name)'' -userPassword_SPO (ConvertTo-SecureString -AsPlainText $(SPO-SvcAcct-Pswd) -Force) `
                -userName_SP ''$(SP-SvcAcct-Name)'' -userPassword_SP (ConvertTo-SecureString -AsPlainText $(SP-SvcAcct-Pswd) -Force) `
                -resultsFolder ''$(System.DefaultWorkingDirectory)\results''  `
                -sourceFolder ''$(filesLocation)'' `
                -environment ''${{ parameters.environment }}'' -buildId ''${{ parameters.environment }} $(Build.BuildNumber)'' '
          displayName: ShareGate Migration

        # Copy results
        - task: CopyFiles@2
          condition: always()
          displayName: 'Archiving migration results'
          inputs:
            contents: |
              $(System.DefaultWorkingDirectory)\results\*
            targetFolder: '$(Build.ArtifactStagingDirectory)'
            flattenFolders: true

        - task: PublishBuildArtifacts@1
          inputs:
            artifactName: '${{ parameters.environment }} ShareGate'
          condition: always()
          displayName: 'Publish Artifact: ShareGate'

It is even possible to use set the ShareGate task name using pipeline's variables.
The -buildId ''${{ parameters.environment }} $(Build.BuildNumber)'' ' parameter is used in PowerShell

Copy-Content  -TaskName "$srcListName to $dstListName ($buildId)"

allowing me to comfortably review the tasks progress from using the ShareGate application: