DEV Community: Mathieu BUISSON

Writing and Using Custom Assertions for Pester Tests

Mathieu BUISSON — Tue, 31 Oct 2017 00:00:00 +0000

Pester, the awesome PowerShell testing framework has recently introduced a capability which allows us to extend our test assertions. This allows to simplify our tests by abstracting custom or complex assertion logic away from the tests and into separate scripts or modules.

Keeping the test scripts as clean and readable as possible is central to leveraging the simplicity and elegance of Pester’s DSL. Also, I really like the idea that test scripts can act as an executable (potentially business-readable) specification.

Concretely, “custom assertions” means that we can plug additional operators into Pester ’s assertion function : Should. Assuming we are using the Pester version 4.0.8, there are quite a few built-in Should operators :

C:\> $PesterPath = (Get-Module -Name 'Pester' -ListAvailable)[0].ModuleBase
C:\> $AllItems = (Get-ChildItem "$PesterPath\Functions\Assertions\").BaseName
C:\> $AllItems.Where({ $_ -notmatch 'Should|Set-' })
Be
BeGreaterThan
BeIn
BeLessThan
BeLike
BeLikeExactly
BeNullOrEmpty
BeOfType
Exist
FileContentMatch
FileContentMatchExactly
FileContentMatchMultiline
Match
MatchExactly
PesterThrow

But in some cases, it is possible that none of these operators fit the type of assertion/comparison needed in our tests.

For example, let’s say we need to validate that a number is within an inclusive range. With the built-in operators we are limited to the BeGreaterThan and BeLessThan, so our test would look like this :

Describe 'Built-in assertions with numbers' {
    It '55.5 should be in range [0-100]' {
        55.5 | Should -BeGreaterThan -0.000001
        55.5 | Should -BeLessThan 100.000001
    }
}

First, it forces us to have 2 assertions in a single test, which is not ideal.

Second, because we want an inclusive range and there is no inclusive operators like BeGreaterThanOrEqualTo or BeLessThanOrEqualTo, we have to modify the “expected” part of the assertion.

Instead of comparing the value to the expected low end of the range (0), we have to compare it with the “expected” minus “just a little bit” to ensure that the assertion passes if the value equals the expected low end of the range. A similar gymnastic is needed when asserting the high end of the range.

This is not only confusing, but also unreliable. The precision of the assertion depends on how small is the “just a little bit”, or in other words, how many decimal places we add/remove to the “expected” part of the assertion.

This whole thing feels wrong so let’s write a better assertion using a custom Should operator.

Writing a custom Should operator

There isn’t much documentation nor examples out there yet, so for now, the best place to start is this example in the Pester wiki.

Our custom operator is going to be named BeInRange and unlike the examples we can find, it will require 2 values to represent the “expected” part of the assertion :

$Min for the low end of the range
$Max for the high end of the range

So here is what the assertion operator function BeInRange looks like :

Function BeInRange {
<#
.SYNOPSIS
Tests whether a value is in a given inclusive range
#>
    [CmdletBinding()]
    Param(
        $ActualValue,
        $Min,
        $Max,
        [switch]$Negate
    )

    [bool]$Pass = $ActualValue -ge $Min -and $ActualValue -le $Max
    If ( $Negate ) { $Pass = -not($Pass) }

    If ( -not($Pass) ) {
        If ( $Negate ) {
            $FailureMessage = 'Expected: value {{{0}}} to be outside the range {{{1}-{2}}} but it was in the range.' -f $ActualValue, $Min, $Max
        }
        Else {
            $FailureMessage = 'Expected: value {{{0}}} to be in the range {{{1}-{2}}} but it was outside the range.' -f $ActualValue, $Min, $Max
        }
    }

    $ObjProperties = @{
        Succeeded = $Pass
        FailureMessage = $FailureMessage
    }
    return New-Object PSObject -Property $ObjProperties
}

First thing to note : the parameter representing the “actual” part of the assertion has to be named ActualValue. If not, Pester ’s internal function Invoke-Assertion blows up because it calls any assertion function with the ActualValue parameter to pass the asserted value.

All Should operators can be negated by inserting -Not before them. For our custom operator to respect this behavior, we need to implement the Negate switch parameter, as seen above.

What’s the deal with the triple braces in the $FailureMessage string ?

Brace #1 : The string formatting operator (-f) uses {} as placeholders

Brace #2 : We enclose the currently asserted values in {} to be consistent with built-in assertion’s failure messages

Brace #3 : The string formatting operator freaks out if the string contains braces, so we double the second set of braces to escape them

Our assertion operator function has a contractual obligation : it has to return an object of the type [PSObject] with a property named Succeeded and a property named FailureMessage. So we built $ObjProperties accordingly.

How do we know that ?

C:\> Get-Help 'Add-AssertionOperator' -Parameter 'Test'

-Test <ScriptBlock>
    The test function. The function must return a PSObject with a [Bool]succeeded
    and a [string]failureMessage property.

    Required? true
    Position? 2
    Default value
    Accept pipeline input? false
    Accept wildcard characters? false

Using a custom Should operator in our tests

We put our BeInRange function in a module named CustomAssertions.psm1 to make it easy to reuse. We could even store the module within a location in our $Env:PSModulePath to allow importing it by name, instead of by path.

There is another way : put the custom operator function in a .ps1 file inside Pester ’s assertions folder : "$((Get-Module Pester).ModuleBase)\Functions\Assertions\".

Pester picks up the operators automatically from this location but this might be less flexible, so choose a method based on your preference/scenario.

Now is the time to write tests to verify how much cleaner our assertions can be with our custom operator and ensure that it behaves as expected :

Import-Module "$PSScriptRoot\CustomAssertions.psm1" -Force
Add-AssertionOperator -Name 'BeInRange' -Test $Function:BeInRange

Describe 'BeInRange assertions with numbers' {
    It '55.5 should be in range [0-100]' {
        55.5 | Should -BeInRange -Min 0 -Max 100
    }
    It '0 should be in range [0-100] (inclusive range)' {
        0 | Should -BeInRange -Min 0 -Max 100
    }
    It '80 should not be in range [0-55.5]' {
        80 | Should -Not -BeInRange -Min 0 -Max 55.5
    }
    It 'Should fail (to verify the failure message)' {
        1 | Should -BeInRange -Min 10 -Max 20
    }
}

The first line imports the module to make our BeInRange function visible in the global scope to ensure Add-AssertionOperator will see it.

Add-AssertionOperator is the function from Pester which enables the magic. It registers a custom assertion operator function with Pester which integrates it with the Should function as a dynamic parameter. Clever stuff.

Unfortunately, as soon as we run this tests script more than once (without unloading the Pester module), we get this error :

Executing script .\CustomAssertions.Tests.ps1
  [-] Error occurred in test script '.\CustomAssertions.Tests.ps1' 35ms
    RuntimeException: Assertion operator name 'BeInRange' has been added multiple times.
    at Assert-AssertionOperatorNameIsUnique, C:\Program Files\WindowsPowerShell\Modules\Pester\4.0.8\Pester.psm1: line 243
    at Add-AssertionOperator, C:\Program Files\WindowsPowerShell\Modules\Pester\4.0.8\Pester.psm1: line 211
    at <ScriptBlock>, C:\CustomAssertions\CustomAssertions.Tests.ps1: line 2
    at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\Pester\4.0.8\Pester.psm1: line 802
    at Invoke-Pester<End>, C:\Program Files\WindowsPowerShell\Modules\Pester\4.0.8\Pester.psm1: line 817

Thankfully this bug is fixed in the master branch and slated for milestone 4.1.

As we can see above, the assertions using the BeInRange operator are simpler and more readable than the initial example using the built-in operators.

Now let’s see if it works :

C:\> $TestsPath = 'C:\CustomAssertions\CustomAssertions.Tests.ps1'
C:\> Invoke-Pester -Script $TestsPath
Executing all tests in 'C:\CustomAssertions\CustomAssertions.Tests.ps1'

Executing script C:\CustomAssertions\CustomAssertions.Tests.ps1

  Describing BeInRange assertions with numbers
    [+] 55.5 should be in range [0-100] 94ms
    [+] 0 should be in range [0-100] (inclusive range) 53ms
    [+] 80 should not be in range [0-55.5] 51ms
    [-] Should fail (to verify the failure message) 38ms
      Expected: value {1} to be in the range {10-20} but it was outside the range.
      15: 1 | Should -BeInRange -Min 10 -Max 20
      at Invoke-Assertion, C:\Program Files\WindowsPowerShell\Modules\Pester\4.0.8\Functions\Assertions\Should.ps1: line 209
      at <ScriptBlock>, C:\CustomAssertions\CustomAssertions.Tests.ps1: line 15
Tests completed in 237ms
Tests Passed: 3, Failed: 1, Skipped: 0, Pending: 0, Inconclusive: 0

This works as expected.

But does it work with objects of the type [string] ? This would allow us to do assertions related to alphabetical ordering.

How about [datetime] objects ? This would allow to validate that a [datetime] value is within an expected date (or time) range.

We add the following tests to our tests script :

Context 'Assertions on [string] objects' {

    It '"abcd" should be in range ["ab"-"yz"]' {
        'abcd' | Should -BeInRange -Min 'ab' -Max 'yz'
    }
    It '"a" should be in range ["a"-"yz"]' {
        'a' | Should -BeInRange -Min 'a' -Max 'yz'
    }
    It '"az" should not be in range ["ab"-"aefg"]' {
        'az' | Should -Not -BeInRange -Min 'ab' -Max 'aefg'
    }
}
Context 'Assertions on [datetime] objects' {
    $Now = [datetime]::Now

    It 'The 1st of October 2017 should be in the range representing the current year' {
        $YearStart = Get-Date -Month 1 -Day 1
        $YearEnd = Get-Date -Month 12 -Day 31
        $FirstOct = Get-Date -Year 2017 -Month 10 -Day 1
        $FirstOct | Should -BeInRange -Min $YearStart -Max $YearEnd
    }
    It 'Today at 10 AM should be between today at 01 AM and now' {
        $DayStart = Get-Date -Hour 1 -Minute 0
        Get-Date -Hour 10 -Minute 0 | Should -BeInRange -Min $DayStart -Max $Now
    }
    It 'Now should not be in the range representing last month' {
        $LastMonthStart = (Get-Date -Day 1 -Hour 0 -Minute 0).AddMonths(-1)
        $LastMonthEnd = (Get-Date -Day 1 -Hour 0 -Minute 0).AddMinutes(-1)
        $Now | Should -Not -BeInRange -Min $LastMonthStart -Max $LastMonthEnd
    }
    It 'Now should not be in a range located in the future' {
        $FutureStart = Get-Date -Year 2117
        $FutureEnd = Get-Date -Year 2217
        $Now | Should -Not -BeInRange -Min $FutureStart -Max $FutureEnd
    }
}

Then, we run the tests script again :

C:\> Invoke-Pester -Script $TestsPath                                                         
Executing all tests in 'C:\CustomAssertions\CustomAssertions.Tests.ps1'                       

Executing script C:\CustomAssertions\CustomAssertions.Tests.ps1                               

  Describing BeInRange assertions with numbers                                                

    Context Assertions on numbers                                                             
      [+] 55.5 should be in range [0-100] 76ms                                                
      [+] 0 should be in range [0-100] (inclusive range) 27ms                                 
      [+] 80 should not be in range [0-55.5] 28ms                                             
      [-] Should fail (to verify the failure message) 28ms                                    
        Expected: value {1} to be in the range {10-20} but it was outside the range.          
        17: 1 | Should -BeInRange -Min 10 -Max 20                                 
        at Invoke-Assertion, C:\Program Files\WindowsPowerShell\Modules\Pester\4.0.8\Functions
\Assertions\Should.ps1: line 209                                                              
        at <ScriptBlock>, C:\CustomAssertions\CustomAssertions.Tests.ps1: line 17             

    Context Assertions on [string] objects                                                    
      [+] "abcd" should be in range ["ab"-"yz"] 65ms                                          
      [+] "a" should be in range ["a"-"yz"] 26ms                                              
      [+] "az" should not be in range ["ab"-"aefg"] 16ms                                      

    Context Assertions on [datetime] objects                                                  
      [+] The 1st of October 2017 should be in the range representing the current year 68ms   
      [+] Today at 10 AM should be between today at 01 AM and now 31ms                        
      [+] Now should not be in the range representing last month 32ms                         
      [+] Now should not be in a range located in the future 33ms                             
Tests completed in 436ms                                                                      
Tests Passed: 10, Failed: 1, Skipped: 0, Pending: 0, Inconclusive: 0

Indeed, it does work for these 3 types of objects.

Now that we know that the “expected” part of the assertion can be comprised of multiple values, this enables complex or specialized assertions. Let’s try a more complex assertion operator which may be useful in infrastructure validation scenarios.

Asserting that an IP address is in a given subnet

We are going to write a new function named BeInSubnet containing logic to check if an IPv4 address is in the same subnet as a specified address (this can be the network ID but it doesn’t have to be) and a specified subnet mask. We’ll add it to our existing CustomAssertions module.

Here is the function :

Function BeInSubnet {
<#
.SYNOPSIS
Tests whether an IPv4 address in the same subnet as a given address with a given subnet mask.
#>
    [CmdletBinding()]
    Param(
        $ActualValue,
        $Network,
        $Mask,
        [switch]$Negate
    )
    If ( $ActualValue -isnot [ipaddress] ) {
        $ActualValue = $ActualValue -as [ipaddress]
    }
    If ( $Network -isnot [ipaddress] ) {
        $Network = $Network -as [ipaddress]
    }
    If ( $Mask -isnot [ipaddress] ) {
        $Mask = $Mask -as [ipaddress]
    }

    $ActualNetworkBinary = $ActualValue.Address -band $Mask.Address
    $ExpectedNetworkBinary = $Network.Address -band $Mask.Address

    [bool]$Pass = $ActualNetworkBinary -eq $ExpectedNetworkBinary
    If ( $Negate ) { $Pass = -not($Pass) }

    If ( -not($Pass) ) {
        $ActualSubnetString = ($ActualNetworkBinary -as [ipaddress]).IPAddressToString
        If ( $Negate ) {
            $FailureMessage = 'Expected: address {{{0}}} to be outside subnet {{{1}}} with mask {{{2}}} but was within it.' -f $ActualValue.IPAddressToString, $Network.IPAddressToString, $Mask.IPAddressToString
        }
        Else {
            $FailureMessage = 'Expected: address {{{0}}} to be in subnet {{{1}}} with mask {{{2}}} but was in subnet {{{3}}}.' -f $ActualValue.IPAddressToString, $Network.IPAddressToString, $Mask.IPAddressToString, $ActualSubnetString
        }
    }

    $ObjProperties = @{
        Succeeded = $Pass
        FailureMessage = $FailureMessage
    }
    return New-Object PSObject -Property $ObjProperties
}

The function’s parameters don’t enforce a specific [type], this is to make the function more flexible. That way, the assertions using this operator will be able to pass [string] or [ipaddress] objects into it.

This is why we check the type of each value passed via the function’s parameters $ActualValue, $Network, $Mask and convert them to the type [ipaddress] for further manipulation.

The rest of the function is fairly similar to our previous custom operator function.

Then, we create a tests script (BeInSubnet.Tests.ps1) to verify that our BeInSubnet custom assertion operator behaves as expected :

Import-Module "$PSScriptRoot\CustomAssertions.psm1" -Force
Add-AssertionOperator -Name 'BeInSubnet' -Test $Function:BeInSubnet

Describe 'BeInSubnet assertions' {
    Context 'Assertions on [string] objects' {

        It '"10.1.5.193" should be in the same subnet as "10.1.5.0" with mask "255.255.255.0"' {
            '10.1.5.193' | Should -BeInSubnet -Network '10.1.5.0' -Mask '255.255.255.0'
        }
        It '"10.1.5.0" should be in the same subnet as "10.1.5.0" with mask "255.255.255.0"' {
            '10.1.5.0' | Should -BeInSubnet -Network '10.1.5.0' -Mask '255.255.255.0'
        }
        It '"10.1.5.193" should be in the same subnet as "10.1.5.0" with mask "255.0.0.0"' {
            '10.1.5.193' | Should -BeInSubnet -Network '10.1.5.0' -Mask '255.0.0.0'
        }
        It '"10.1.5.193" should not be in the same subnet as "10.1.5.0" with mask "255.255.255.128"' {
            '10.1.5.193' | Should -Not -BeInSubnet -Network '10.1.5.0' -Mask '255.255.255.128'
        }
        It 'Should fail (to verify the failure message)' {
            '10.1.5.193' | Should -BeInSubnet -Network '10.1.5.0' -Mask '255.255.255.128'
        }
    }
    Context 'Assertions on [ipaddress] objects' {
        $Value = '10.1.5.193' -as [ipaddress]
        $Network = '10.1.5.0' -as [ipaddress]
        $SubnetMask = '255.255.255.0' -as [ipaddress]
        $LargeSubnetMask = '255.0.0.0' -as [ipaddress]
        $SmallSubnetMask = '255.255.255.128' -as [ipaddress]

        It '"10.1.5.193" should be in the same subnet as "10.1.5.0" with mask "255.255.255.0"' {
            $Value | Should -BeInSubnet -Network $Network -Mask $SubnetMask
        }
        It '"10.1.5.0" should be in the same subnet as "10.1.5.0" with mask "255.255.255.0"' {
            $Network | Should -BeInSubnet -Network $Network -Mask $SubnetMask
        }
        It '"10.1.5.193" should be in the same subnet as "10.1.5.0" with mask "255.0.0.0"' {
            $Value | Should -BeInSubnet -Network $Network -Mask $LargeSubnetMask
        }
        It '"10.1.5.193" should not be in the same subnet as "10.1.5.0" with mask "255.255.255.128"' {
            $Value | Should -Not -BeInSubnet -Network $Network -Mask $SmallSubnetMask
        }
        It 'Should fail (to verify the failure message)' {
            $Value | Should -BeInSubnet -Network $Network -Mask $SmallSubnetMask
        }
    }
}

The tests are clean and readable. Now, imagine if we had to do the same assertions with the built-in Should operators… The tests script would have been cluttered with a large amount of [ipaddress] manipulation code, unless this code was extracted into a helper function.

Let’s run these tests and check the result :

C:\> $SubnetTestsPath = 'C:\CustomAssertions\BeInSubnet.Tests.ps1'
C:\> Invoke-Pester -Script $SubnetTestsPath
Executing all tests in 'C:\CustomAssertions\BeInSubnet.Tests.ps1'

Executing script C:\CustomAssertions\BeInSubnet.Tests.ps1

  Describing BeInSubnet assertions

    Context Assertions on [string] objects
      [+] "10.1.5.193" should be in the same subnet as "10.1.5.0" with mask "255.255.255.0" 138ms
      [+] "10.1.5.0" should be in the same subnet as "10.1.5.0" with mask "255.255.255.0" 20ms
      [+] "10.1.5.193" should be in the same subnet as "10.1.5.0" with mask "255.0.0.0" 23ms
      [+] "10.1.5.193" should not be in the same subnet as "10.1.5.0" with mask "255.255.255.128" 23ms
      [-] Should fail (to verify the failure message) 27ms
        Expected: address {10.1.5.193} to be in subnet {10.1.5.0} with mask {255.255.255.128} but was in subnet {10.1.5.128}.
        20: '10.1.5.193' | Should -BeInSubnet -Network '10.1.5.0' -Mask '255.255.255.128'
        at Invoke-Assertion, C:\Program Files\WindowsPowerShell\Modules\Pester\4.0.8\Functions\Assertions\Should.ps1: line 209
        at <ScriptBlock>, C:\CustomAssertions\BeInSubnet.Tests.ps1: line 20

    Context Assertions on [ipaddress] objects
      [+] "10.1.5.193" should be in the same subnet as "10.1.5.0" with mask "255.255.255.0" 80ms
      [+] "10.1.5.0" should be in the same subnet as "10.1.5.0" with mask "255.255.255.0" 26ms
      [+] "10.1.5.193" should be in the same subnet as "10.1.5.0" with mask "255.0.0.0" 28ms
      [+] "10.1.5.193" should not be in the same subnet as "10.1.5.0" with mask "255.255.255.128" 18ms
      [-] Should fail (to verify the failure message) 29ms
        Expected: address {10.1.5.193} to be in subnet {10.1.5.0} with mask {255.255.255.128} but was in subnet {10.1.5.128}.
        43: $Value | Should -BeInSubnet -Network $Network -Mask $SmallSubnetMask
        at Invoke-Assertion, C:\Program Files\WindowsPowerShell\Modules\Pester\4.0.8\Functions\Assertions\Should.ps1: line 209
        at <ScriptBlock>, C:\CustomAssertions\BeInSubnet.Tests.ps1: line 43
Tests completed in 418ms
Tests Passed: 8, Failed: 2, Skipped: 0, Pending: 0, Inconclusive: 0

Good, everything works as expected.

Also, we can see that the failure message is quite helpful. It tells us exactly what is going on, with the “actual” values and the “expected” values. This is a major advantage of custom assertions, it enables us to create more meaningful failure messages because these messages can be tailored for specific assertions.

Conclusion

While there are a few quirks to iron out to make this feature fully usable and mature, the ability to extend Pester ’s assertion operators with simple PowerShell functions is very powerful.

This allows to perform very complex or specialized assertions in our tests while keeping them relatively human-readable and remaining true to Pester ’s DSL. This expands Pester ’s assertions capabilities, as well as its use cases.

Don’t go too crazy, though. I would advise keeping the complexity of the assertion logic to a minimum, because the more complexity, the more chance that something may not work completely as expected. And we want assertions to be very reliable , we want tests to fail because the code under test doesn’t behave as intended, not the assertion logic.

Making PSScriptAnalyzer a first-class citizen in a PowerShell CI pipeline

Mathieu BUISSON — Wed, 30 Nov 2016 00:00:00 +0000

As you already know if you have read this or this, I’m a big fan of PSScriptAnalyzer to maintain a certain coding standard. This is especially powerful inside a release pipeline because this allows us to enforce that coding standard.

In our CI pipeline, we can easily make the build fail if our code violates PSScriptAnalyzer rule(s).

That’s great, but the main point of continuous integration is to give quick feedback to developers about their code change(s). It is about catching problems early to fix them early. So the question is :

How can we make our CI tool publish PSScriptAnalyzer results with the information we need to remediate any violation ?

All CI tools have ways to publish test results to make them highly visible, to drill down into a test failure and do some reporting.

Since we are talking about a PowerShell pipeline, we are most likely already using Pester to test our PowerShell code.

Pester can spit out results in the same XML format as NUnit and these NUnit XML files can be consumed and published by most CI tools.

It makes a lot of sense to leverage this Pester integration as a universal CI glue and run our PSScriptAnalyzer checks as Pester tests.

Let’s look at possible ways to do that.

One Pester test checking PSScriptAnalyzer output

Here is probably the simplest way to invoke PSScriptAnalyzer from Pester :

Describe 'PSScriptAnalyzer analysis' {    
    $ScriptAnalyzerResults = Invoke-ScriptAnalyzer -Path '.\Example.ps1' -Severity Warning

    It 'Should not return any violation' {
        $ScriptAnalyzerResults | Should BeNullOrEmpty
    }
}

Here, we are checking all the rules which have a Warning severity within one single test.

Then, we rely on the fact that if PSScriptAnalyzer returns something, it means that they were at least one violation and if PSScriptAnalyzer returns nothing, it’s all good.

There are 2 problems here :

We are evaluating a whole bunch of rules in a single test, so the test name cannot tell us which rule was violated
If there are more than one violation, the Pester message gives us useless information

How useless ? Well, let’s see :

Invoke-Pester -Script '.\Example.Tests.ps1'
Executing all tests in .\Example.Tests.ps1

Executing script .\Example.Tests.ps1

  Describing PSScriptAnalyzer analysis
    [-] Should not return any violation 1.52s
      Expected: value to be empty but it was {Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord}
      at <ScriptBlock>, E:\Example.Tests.ps1: line 5
      5: $ScriptAnalyzerResults | Should BeNullOrEmpty
Tests completed in 1.52s
Tests Passed: 0, Failed: 1, Skipped: 0, Pending: 0, Inconclusive: 0

The Pester failure message gives us the object type of the PSScriptAnalyzer results, instead of their contents.

One Pester test per PSScriptAnalyzer rule

This is a pretty typical (and better) way of running PSScriptAnalyzer checks via Pester.

Describe 'PSScriptAnalyzer analysis' {    
    $ScriptAnalyzerRules = Get-ScriptAnalyzerRule -Name "PSAvoid*"

    Foreach ( $Rule in $ScriptAnalyzerRules ) {
        It "Should not return any violation for the rule : $($Rule.RuleName)" {
            Invoke-ScriptAnalyzer -Path ".\Example.ps1" -IncludeRule $Rule.RuleName |
            Should BeNullOrEmpty
        }
    }
}

In this case, the first step is to get a list of the rules that we want to evaluate. Here, I changed the list of rules to : all rules which have a name starting with PSAvoid. This is just to show that we can filter the rules by name, as well as by severity.

Then, we loop through this list of rules and have a Pester test evaluating each rule, one by one. As we can see below, the output is much more useful :

This is definitely better but we still encounter the same issue as before because there were more than one violation for that PSAvoidUsingWMICmdlet rule. So we still don’t get the file name and the line number.

We could use a nested loop : for each rule, we would loop through each file.

That would be more granular and reduce the risk of this particular issue. But if a single file violated the same rule more than once, we would still have the same problem.

So, I decided to go take a different route to address this problem : taking the output from PSScriptAnalyzer and converting it to a test result file, using the same XML schema as Pester and NUnit.

Converting PSScriptAnalyzer output to a test result file

For that purpose, I wrote a function named Export-NUnitXml, which is available in this module.

Here are the high-level steps of what Export-NUnitXml does :

Take the output of PSScriptAnalyzer as its input
Create an XML document containing a “test-case” node for each input object(s)
Write this XML document to the file specified via the Path parameter

Here is an example of how we can use this within a build script (in Appveyor, in this case) :

$ScriptAnalyzerRules = Get-ScriptAnalyzerRule -Severity Warning
$ScriptAnalyzerResult = Invoke-ScriptAnalyzer -Path '.\CustomPSScriptAnalyzerRules\Example.ps1' -IncludeRule $ScriptAnalyzerRules
If ( $ScriptAnalyzerResult ) {  
    $ScriptAnalyzerResultString = $ScriptAnalyzerResult | Out-String
    Write-Warning $ScriptAnalyzerResultString
}
Import-Module '.\Export-NUnitXml\Export-NUnitXml.psm1' -Force
Export-NUnitXml -ScriptAnalyzerResult $ScriptAnalyzerResult -Path '.\ScriptAnalyzerResult.xml'

(New-Object 'System.Net.WebClient').UploadFile("https://ci.appveyor.com/api/testresults/nunit/$($env:APPVEYOR_JOB_ID)", '.\ScriptAnalyzerResult.xml')
If ( $ScriptAnalyzerResult ) {        
    # Failing the build
    Throw 'There was PSScriptAnalyzer violation(s). See test results for more information.'
}

And here is the result in Appveyor :

Just by reading the name of the test case, we get the essential information : the rule name, the file name and even the line number.

Also, we can expand any failed test to get additional information.

For example, the last 2 tests are expanded below :

The “ Stacktrace ” section provides additional details, like the rule severity and the actual offending code. Another nice touch is that the “ Error message ” section gives us the rule message, which normally provides an actionable recommendation to remediate the problem.

But, what if PSScriptAnalyzer returns nothing ?

Export-NUnitXml does handle this scenario gracefully because its ScriptAnalyzerResult parameter accepts $Null. In this case, the test result file will contain only 1 test case and this test will pass.

So now, we not only have quick feedback on our adherence to coding standards, but we also get actionable guidance on how to improve.

And remember, this NUnit XML format is widely supported in the CI/CD tooling world, so this would work similarly in TeamCity, Microsoft VSTS, and others…

This post was originally published on mathieubuisson.github.io