PowerShell, Files and Azure Storage Account blobs

Azure Storage Blob is a cost-effective solution to store any kind of file in the cloud. It's secure, reliable and resilient. More, Blob can be used as a trigger for Azure Functions when you add a file and you can play with different pricing tiers.
You can use it with an application, for your deployment in Azure or to archive files from your file servers.

You have several options to send files to a blob container, you can use databox if you need to archive Tb of files or databox edge to links your files servers to Azure, you can use data explorer to copy files. But what can you do if you need to archive files based on a pattern and keep the folder structure?

Let says that we have a local folder with several files. You want to upload only jpg files older than 2 weeks and preserve the folder structure inside the container. The process needs to run every day or every week.

To perform these operations, I use PowerShell core and the cross-platform module for Azure AZ

In the first step, we need to create a storage account and a blob container.

Remember, storage account in Azure must have a globally unique name. I use a random string to create the storage name and I use the tag DisplayName for a human reading name.

$LowercaseLetterCodes = (97..122)

$StorageAccountName = -join ((Get-Random -InputObject $LowercaseLetterCodes -Count 24) | Foreach-Object {[char]$_} ) 


$StorageAccountName = -join ((Get-Random -InputObject $LowercaseLetterCodes -Count 24) | Foreach-Object {[char]$_} ) 

Now we need to create a blob container

New-AzStorageContainer -Name "archive" -Context $AzStorageAccount.Context -Permission off | out-null 

And we need a SAS token key

$AzStorageSasToken = New-AzStorageContainerSASToken -container "archive" -Permission rwdl -ExpiryTime (get-date).AddMonths(3) -Context $AzStorageAccount.Context 

This key will be used later to upload files to the blob container.

In the second step, we need to select the files we want to upload.

Remember we need to get JPG files older than 2 weeks. To only list JPG files. We can use the –Filter. It's the more efficient way to achieve the goal. But the param only takes a single string and only support * and ?

JPG files can have at least two extensions, jpeg, and jpg. The filter param can be designed like this:

-filter "*.jp*g" 

We need to scan the whole directory and not only the current folder, so we need to use the –recurse switch.

We can also use the –Attributes parameter to exclude directory, temporary and encrypted files.

-Attributes !Directory+!System+!Encrypted+!Temporary 

There is no direct way to filter files based on a date. The only way is to filter the output. In PowerShell, it means to use the Pipeline By default PowerShell send objects to the pipeline as they are generated. Most of the time you don't have to care about that, but for a large collection, you can manage buffer OutBuffer. Objects will be sent to the pipeline as a batch.

The size of the buffer depends on the size of the batch, I choose 100 because I wanted less than 5 % of the number of files in the folders.

To prevent memory grow, we need to limit the child-item output by using select-object. We only need the path, the name and the length of each file

 | select-object FullName,Name,Length 

Finally, we can filter the result, from the file object we can use the LastWriteTime metadata to only get files older than 2 weeks

| where-object LastWriteTime -lt $dateToday.AddDays(-15) 

Here's the final command

$FilesListObject = get-childitem -Path $Path -Attributes !Directory+!System+!Encrypted+!Temporary -Recurse -filter "*.jp*g" -ErrorAction SilentlyContinue | select-object FullName,Name,LastWriteTime,Length -OutBuffer 100 | where-object LastWriteTime -lt $dateToday.AddDays(-15) 

Now we can start to send the file to Azure. We need the AZ Module, the Storage account name, the container name and the SAS Token created in the first step.

First things we need to create an Azure Storage Context

$AzureStorageContext = New-AzStorageContext $StorageAccountName -SasToken $AzStorageSasToken 

We need to preserve the folder structure. For each file we need to recreate the relative path to the main folder.

Our main folder is in the $Path variable and the $FilesListObject contain the name and the full path for each file. We just need to subtract the $Path to the Full Path of each file.

foreach ($FileInfo in $FilesListObject) {  

$FileRelativePath = Split-Path -Path $FileInfo.FullName 

$VirtualPath = (Split-Path $path -Leaf) + ($FileRelativePath.Substring($Path.Length, ($FileRelativePath.Length - $Path.Length))) + "\" + $FileInfo.Name 

} 

We will use this path with the –blob parameter of the set-azStorageBlobContent but I have a little problem. French people love to put accents everywhere (we also miss H and S, but it’s another story). I need some kind of normalization.

We can use System.textNormalization and a string builder

[System.Text.NormalizationForm]$NormalizationForm = "FormD" 

$Normalized = ($FileInfo.Name).Normalize($NormalizationForm) 

$NormalizedFileName = New-Object -TypeName System.Text.StringBuilder 





$normalized.ToCharArray() | ForEach-Object -Process { 

if ([Globalization.CharUnicodeInfo]::GetUnicodeCategory($psitem) -ne [Globalization.UnicodeCategory]::NonSpacingMark) 

{ 

[void]$NormalizedFileName.Append($psitem) 

} 

} 

Now we can send files to Azure with the set-azStorageBlobContent

foreach ($FileInfo in $FilesListObject) {  

  [System.Text.NormalizationForm]$NormalizationForm = "FormD" 

  $Normalized = ($FileInfo.Name).Normalize($NormalizationForm) 

  $NormalizedFileName = New-Object -TypeName System.Text.StringBuilder 

  $normalized.ToCharArray() | ForEach-Object -Process { 

  if ([Globalization.CharUnicodeInfo]::GetUnicodeCategory($psitem) -ne [Globalization.UnicodeCategory]::NonSpacingMark) 

  { 

      [void]$NormalizedFileName.Append($psitem) 

  } 

} 

   $FileRelativePath = Split-Path -Path $FileInfo.FullName 

   $VirtualPath = (Split-Path $path -Leaf) + ($FileRelativePath.Substring($Path.Length, ($FileRelativePath.Length - $Path.Length))) + "\" + $NormalizedFileName.ToString() 





   Set-AzStorageBlobContent -File $FileInfo.FullName -Container "archive" -Context 
   $AzureStorageContext -Blob $VirtualPath -Force 

}