loading...

Setup your Azure Search with PowerShell

manuelsidler profile image Manuel Sidler ・4 min read

Microsoft provides cmdlets through Azure PowerShell and a REST API to manage your Azure Search resource.

In this scenario we need a Azure Search service which will:

  • extract storage metadata from an Azure Blob container
  • store the metadata inside an index
  • provide a custom key to query our index

To fullfil these requirements we're going to use a combination of the Azure PowerShell cmdlets and REST API endpoints to setup an Azure Search with the following components:

  • Azure Search service
  • Search query key
  • Search index connected to an Azure Blob storage

In order to achieve this, we're going to take these steps:

  1. Define common variables
  2. Login to Azure
  3. Create search service
  4. Create search query key
  5. Get search primary admin key
  6. Create index
  7. Create data source
  8. Create indexer

Prerequisites

Variables

First of all we need to define some variables which we'll use throughout the upcoming steps.

$tenantId = "<tenantId>" # eg. 12133905-3bce-414e-983f-79013911deee
$subscriptionId = "<subscriptionId>" # eg. 510f6c5b-b136-45e7-b239-dccb600331ea
$resourceGroup = "<resourceGroup>" # eg. my-project
$location = "<location>" # eg. WestEurope
$storageAccountName = "<storageaccount>" # eg. myprojectstorage
$searchName = "<searchName>" # eg. my-project-search

Login

To get access to our Azure resources we need to connect to our Azure account:

Connect-AzAccount -Tenant $tenantId -Subscription $subscriptionId

This cmdlet will open a popup where you can enter your credentials for your Azure account.

Create search service

Assuming the login was successful we're now able to create the search service:

$sku = "Free" # Free|Basic|S1|S2|S3|S3-HD
New-AzSearchService -Name $searchName -ResourceGroupName $resourceGroup -Sku $sku -Location $location

Create search query key

We also need an additional search query key in our scenario:

$queryKey = "MyQueryKey"
New-AzSearchQueryKey -Name $queryKey -ServiceName $searchName -ResourceGroupName $resourceGroup

Get search primary admin key

Unfortunately Azure PowerShell doesn't provide cmdlets to manage your search indexes, indexers and data sources. To create these resources we'll put the Azure REST API into action. But first we need to fetch the search primary admin key which we're going to use as an API key:

$adminKeyPair = Get-AzSearchAdminKeyPair -ResourceGroupName $resourceGroup -ServiceName $searchName
$primaryAdminKey = $adminKeyPair.Primary

Create index

Instead of using a variable for the index definition, I would like to store it as JSON inside a folder. Once we stored the index definition, we can iterate through them when actually calling the Create Index endpoint. We're going to keep this index as simple as possible. Fields:

  • ID (Primary Key)
  • Title
  • Tags (Filterable Collection)
  • metadata_storage_path (Path of the Blob)

/indexes/myindex.json

{
    "name": "my-index",  
    "fields": [
        {"name": "ID", "type": "Edm.String", "key": true, "filterable": false, "sortable": false, "facetable": false, "searchable": false},
        {"name": "TITLE", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false},
        {"name": "TAGS", "type": "Collection(Edm.String)", "filterable": true, "sortable": false, "facetable": false, "searchable": false},
        {"name": "metadata_storage_path", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false}
    ]
}

With our definition we're now able to create our index:

Get-ChildItem "indexes" -Filter *.json |
        ForEach-Object {
            $indexDefinition = Get-Content $_.FullName -Raw

            $headers = @{
                'api-key' = $primaryAdminKey
                'Content-Type' = 'application/json' 
                'Accept' = 'application/json' }

            $url = "https://$searchName.search.windows.net/indexes?api-version=2017-11-11"

            Invoke-RestMethod -Uri $url -Headers $headers -Method Post -Body $indexDefinition | ConvertTo-Json
        }

Create data source

We take the same approach for creating the data source. But in this case we need to set the connection string from our storage account. As we're good developers we won't store the string inside the data source JSON definition ;-)

/datasources/mydatasource.json

{   
    "name" : "my-datasource",
    "type" : "azureblob",
    "credentials" : { 
        "connectionString" : "[STORAGECONNECTION]" 
    },  
    "container" : { 
        "name" : "myblobcontainer" 
    }
}

To replace the STORAGECONNECTION placeholder we first need to fetch the connection string:

$storageKey = (Get-AzStorageAccountKey -Name $storageAccountName -ResourceGroupName $resourceGroup)[0].Value

$storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=$storageAccountName;AccountKey=$storageKey;EndpointSuffix=core.windows.net"

With the storage connection string available we now can iterate through the data source folder and use the Create Data Source endpoint to create the data source:

Get-ChildItem "datasources" -Filter *.json |
        ForEach-Object {
            $datasourceDefinition = (Get-Content $_.FullName -Raw).replace("[STORAGECONNECTION]", $storageConnectionString)

            $headers = @{
                'api-key' = $primaryAdminKey
                'Content-Type' = 'application/json' 
                'Accept' = 'application/json' }

             $url = "https://$searchName.search.windows.net/datasources?api-version=2017-11-11"

             Invoke-RestMethod -Uri $url -Headers $headers -Method Post -Body $dataSourceDefinition | ConvertTo-Json
        }

Create indexer

Last but not least we have to create an indexer. Again we use a JSON file to define the indexer. What's important here is that in our scenario we just want to index the storage metadata. You could also choose to extract all metadata or the file content.

/indexers/myindexer.json

{
    "name" : "my-indexer",
    "dataSourceName" : "my-datasource",
    "targetIndexName" : "my-index",
    "schedule" : {
        "interval" : "PT1H"
    },
    "parameters" : {
        "configuration" : {
            "dataToExtract" : "storageMetadata"
        }
     }
}

To create the indexer we're going to use the Create Indexer endpoint:

Get-ChildItem "indexers" -Filter *.json |
        ForEach-Object {
            $indexerDefinition = Get-Content $_.FullName -Raw

            $headers = @{
                'api-key' = $primaryAdminKey
                'Content-Type' = 'application/json' 
                'Accept' = 'application/json' }

            $url = "https://$searchName.search.windows.net/indexers?api-version=2017-11-11"

            Invoke-RestMethod -Uri $url -Headers $headers -Method Post -Body $indexerDefinition | ConvertTo-Json
        }

Conclusion

We now have an Azure Search service which:

  • provides a custom key to query our index
  • extracts storage metadata from an Azure Blob container
  • stores the metadata inside an index

search query

Posted on by:

manuelsidler profile

Manuel Sidler

@manuelsidler

Developer, developer, developer :-)

Discussion

markdown guide