So, previously I wrote about deployment process to IIS servers. However wouldn't it be nice to keep that IIS configuration in sync, moreover between production and staging environment?
Configuration deployment world.
One way is to create the initial installation and configuration script. Nice and fast to setup additional server, a great first step. But:
- The configuration has to be maintained and your script has to account for that (if this already exists then do nothing else configure).
- You need to have a mechanism to run the script. Scheduler?
- Your systems may drift if some processes or users do some configuration.
- You do not know the diff between the current system configuration and the script.
There are many tools that may be used to provide uniform server configuration, like Group Policy, System Center Configuration Manager/Microsoft Endpoint Configuration Manager, PowerShell DSC (Desired State Configuration), Docker and various other software configuration management tools etc. All of those could be used to help achieve completely or partially uniform configuration across multiple hosts. Some of them are not free, some of them depends on Active Directory, some of them have GUI and so on.
Enter PowerShell DSC
Lets just explore PowerShell DSC, that you don't have to license and if you, like me, enjoy using PowerShell, this may be a tool for you.
Things I immediately found satisfying with PowerShell DSC:
- It's PowerShell (with a little bit of additional syntax to learn) and no additional tools to install (apart from PowerShell modules).
- It works over WinRM
- It helps you define Infrastructure-as-Code (https://en.wikipedia.org/wiki/Infrastructure_as_code)
- You can either make report configuration drifts, check them manually or make your host to automatically re-apply whatever was instructed in DSC script.
In an ideal world, once our server is joined to domain, I would like a one-click install for servers, be it staging or production environment. PowerShell DSC achieves quite a lot, but not necessarily a single click deployment. We will see why and how to overcome it.
Why did I go out to try PS DSC?
- Didn't want to write configuration documentation - let there be text that can't be outdated.
- IaC - wanted to have valid configuration in source control
- I heavily use PowerShell and this would be a "natural extension".
- This has come as a bonus: On deployment, check if IIS webserver configuration is as written.
Writing DSC script
I will be using PowerShell 5.1 with DSC 1.1. Within v2, DSC has been split out from powershell package and v3 will provide cross-platform capabilities.
What I'm going to concentrate on is overcoming hurdles when writing DSC and not the actual configuration itself. You can drill down the concepts and how-to documentation to learn DSC. My requirements were:
- I want ONE or maybe two simple commands I can use to configure IIS servers
- I want to be able target all or subset of servers
- I want to target different environments (prod, staging)
Let's look at a simple dsc.ps1 script.
configuration SimpleConfiguration {
# DSC provides only simple tasks. Usually modules will be imported to do additional tasks.
Import-DscResource -ModuleName 'ComputerManagementDsc' -Name 'PendingReboot', 'TimeZone', 'SystemLocale'
# One can evaluate expressions to get the node list E.g: $AllNodes.NodeName - AllNodes can be passed when calling function.
node ("Node1","Node2","Node3")
{
# Call Resource Provider E.g: WindowsFeature, File
WindowsFeature WebServerRole { Ensure = "Present"; Name = "Web-Server" }
SystemLocale EnglishLocale { SystemLocale = "en-US" ; IsSingleInstance = "Yes" }
File site1Folder { DestinationPath = "$env:SystemDrive\inetpub\sites\site1"; Type = 'Directory' }
}
}
Understanding resources
What does SystemLocale EnglishLocale { SystemLocale = "en-US" ; IsSingleInstance = "Yes" } mean?
It means I'm calling SystemLocale resource, giving it whatever friendly name EnglishLocale and passing hashtable with properties resource accepts: SystemLocale, IsSingleInstance. You can get available properties like that:
Get-DscResource SystemLocale -Syntax
SystemLocale [String] #ResourceName
{
IsSingleInstance = [string]{ Yes }
SystemLocale = [string]
[DependsOn = [string[]]]
[PsDscRunAsCredential = [PSCredential]]
}
Why can't it be a simple install script?
It can, but in real world - without the word "simple". It's written in Desired State Configuration Overview for Engineers. In short: you focus on WHAT to do not on HOW. DSC provides idempotent (repeatable) deployments. Along with goodies like Continue configuration after reboot, Credential encryption and other goodies.
Why this silly syntax?
Well, because this resource under the hood actually implements 3 functions: Get, Set, Test
-
Get is a representation of state when, for example, running
Test-DscConfiguration -DetailedorGet-DscConfiguration - Set - Applies configuration
- Test - Determines whether configuration has to be applied. If not, Set is NOT run. That is, if system is already compliant with this resource, it won't re-run.
When Custom resources gets written, these functions get implemented. For example, using built-in Script resource, you also have to implement all of them. Take for example a configuration, that ensures BUILTIN\Users have no access to particular folder:
Script site1FolderPermissions {
GetScript = { @{Result="Path = $env:SystemDrive\inetpub\sites\site1; Acl = $(Get-NTFSAccess "$env:SystemDrive\inetpub\sites\site1" | %{ $_.ToString() })" } }
TestScript = {
(Get-NTFSAccess "$env:SystemDrive\inetpub\sites\site1" | ? Account -eq 'BUILTIN\Users' | measure | select -ExpandProperty Count) -eq 0
}
SetScript = {
Disable-NTFSAccessInheritance -Path "$env:SystemDrive\inetpub\sites\site1"
$acl = Get-Acl "$env:SystemDrive\inetpub\sites\site1"
$acl.Access | ?{ $_.IsInherited -eq $false -and $_.IdentityReference -eq 'BUILTIN\Users' } | % {$acl.RemoveAccessRule($_)}
Set-Acl -AclObject $acl "$env:SystemDrive\inetpub\sites\site1"
}
DependsOn = "[File]site1Folder"
}
Drawbacks
Modules
Drawbacks or pain-points or maybe stuff that hasn't been provided by DSC out-of-the-box: Distributing modules. And this is the reason I can't have a single command to bootstrap my servers. The modules have to be installed on the host that are building the configuration AND the host that gets configuration deployed. Otherwise, when encountering Import-DscResource -Module CertificateDsc -ModuleVersion 5.1.0, it will tell you:
Could not find the module '<CertificateDsc, 5.1.0>'.
However moving Import-DscResource command just will result in error:
Import-DscResource cannot be specified inside of Node context
So the takeaway: we can't install a module and use its resources in a single configuration. These modules are required on the host that are building configuration and on target nodes. Modules are essential to any configuration, except the most trivial ones.
Moreover, the PackageManagement resource you see in a screenshot is a module itself that must be installed from PowerShell gallery before I can actually use it to install other modules 🤦♂️
Remember me telling how DSC enables writing more simple configuration comparing to install scripts? Turns out with DSC comes complexity anyways if I want automatic module provisioning. 🤷♂️ I don't want anyone (and my future self) running the deployment having to think about what modules must be installed on source & target machines.
So I will be using another configuration to install modules, but without any dependencies on modules. I'll run this configuration on hosts that require these modules and only then apply main configuration. First, I apply InstallPSModules configuration and then the SampleConfig configuration. There is a gotcha: I need modules on localhost too. But running Start-DscConfiguration .\InstallPSModules\ -ComputerName localhost gives me error:
The WinRM client sent a request to the remote WS-Management service and was notified that the request size exceeded the configured MaxEnvelopeSize quota
It may be misleading. If I pass -Credential parameter, specifying which user I want to use to connect to localhost (admin), then it works. This is due to the way users/computers/winrm are configured in my environment.
The other thing is that it may override some other DSC configuration applied to this computer. Anyways, I had to choose a tradeoff. I have a build task (a function) that installs modules on local computer and another for deploying modules to target nodes. But then I have to remember adding any dependencies in 2 places. Maybe I'm trying too hard for striving to reach the goal of "1 simple command to perform deployment"...
What I'm doing is installing from PowerShell gallery. You could set-up so that you have modules available on some share/local machine and copy them to target machine C:\Windows\system32\WindowsPowerShell\v1.0\Modules folder. Or perhaps any of these folders:
PS C:\Repos\psdsctest> C:\Tools\PSTools\PsExec64.exe -s powershell.exe -Command "`$env:PSModulePath -split ';'"
C:\Windows\system32\config\systemprofile\Documents\WindowsPowerShell\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules
That's PSModulePath for SYSTEM account. The configuration itself is being applied in the context of SYSTEM account on target nodes. Important to keep in mind when writing configurations. If you need to apply some stuff within context of another user, read about Credential Options in Configuration Data.
Here is an example build task for some configuration I wrote that copies files to particular computers before doing deployment (err, read "Hooking it all up" to see that I'm using a build tool, thus another "weird" syntax):
task copyRequiredFilesToNodes {
exec {
Write-Host "Copying $BuildRoot\winlogbeat to target nodes"
$nodeSessions = New-PSSession -ComputerName $Nodes
Invoke-Command -Session $nodeSessions { Remove-Item -Recurse -Path "C:\Windows\Temp\winlogbeat" -Verbose -ErrorAction SilentlyContinue}
$nodeSessions | % { Copy-Item -Recurse -Path "$BuildRoot\winlogbeat" -Destination "C:\Windows\Temp\" -ToSession $_ -Force -Verbose }
}
}
DSC Engine configuration for enabling reboot
Moreover, DSC Engine itself must be configured to, for example, allow reboots with PendingReboot resource. DSC Engine configuration is applied with Set-DscLocalConfigurationManager command and uses different configuration:
[DSCLocalConfigurationManager()]
configuration DSCConfig
{
Node $AllNodes.NodeName
{
Settings
{
#allow reboot with PendingReboot resource from ComputerManagementDsc module
RebootNodeIfNeeded = $true
}
}
}
Applying configuration:
Set-DscLocalConfigurationManager .\DSCConfig\ -ComputerName localhost -Verbose
So, multiple gotchas here. Lets hook it all up within a single build script we can call.
Targeting other environments
Configuration scripts are powershell scripts. You may choose to pass parameter or detect which env you are in any other way:
$environment = if ($env:USERDOMAIN -ieq "prod.example.com") {"production"} else {"staging"}
Then you can use if statements or override variables based on environment.
if ($environment -eq "production") {
#Call some resources
} else {
#Call other resources
}
Hooking it all up
So I need to take a step back and Start-DscConfiguration won't be my entry point to deployment. I need a .ps1 script that installs modules. Or, bear with me, I'll be using a PowerShell build tool: Invoke-Build.
nightroman
/
Invoke-Build
Build Automation in PowerShell
Build Automation in PowerShell
Invoke-Build is a build and test automation tool which invokes tasks defined in PowerShell v2.0+ scripts. It is similar to psake but arguably easier to use and more powerful. It is complete, bug free, well covered by tests.
In addition to basic task processing the engine supports
- Incremental tasks with effectively processed inputs and outputs.
- Persistent builds which can be resumed after interruptions.
- Parallel builds in separate workspaces with common stats.
- Batch invocation of tests composed as tasks.
- Ability to define new classes of tasks.
Invoke-Build v3.0.1+ is cross-platform with PowerShell Core.
Invoke-Build can be effectively used in VSCode and ISE.
Several PowerShell Team projects use Invoke-Build.
The package
The package includes the engine, helpers, and help:
- Invoke-Build.ps1 - invokes build scripts, this is the build engine
- Build-Checkpoint.ps1 - invokes persistent builds using the engine
- Build-Parallel.ps1 - invokes parallel builds using the engine
- Resolve-MSBuild.ps1 - finds…
You may use other tools too: psake, make, cake, fake or any other *ake you are familiar with. I look at them as a tools that make build tasks behind simple commands and help me answer: How did I run that code again?
So, we still must install Invoke-Build? Well, conveniently, there is a template for invoke-build file that detects if Invoke-Build is installed or not and installs it and we'll use it.
The result ended up using 4 scripts. However it can be used as an example/bootstrap project to speed up your implementation, if you choose to go down the same path:
-
build.ps1- Invoke-Build script. Entry point -
DSCConfig.ps1- DSC Engine configuration. Allows reboots -
InstallPSModules.ps1- Module deployment -
SampleConfig.ps1- Where I would put my configuration.
In the end, I can checkout the code and run:
.\build.ps1 deploy -Nodes localhost
# or
.\build.ps1 deploy -Nodes server1, server2, server3 -Credential (Get-Credential)
# By the way, whatever credential you pass to the build script is only for connecting to target node and starting DSC configuration. Remember that resources itself are applied within SYSTEM context.
I'v put up the scripts at psdsctest repository:
PowerShell DSC - Infrastructure as Code. Work-around pain points.
Show how to initiate PowerShell DSC configuration with one-single command. Includes configuration of DSC Engine and module dependency installation.
Available tasks:
./build.ps1 ?
Name Jobs Synopsis
---- ---- --------
deployDSCConfig {buildDSCConfig, {}} Apply DSC engine configuration to allow reboots. Kind of a special case: https://docs.mi...
installModules {} Install required powershell modules for current host for build to work.
buildDSCConfig {} Generate .mof files configuring local DSC settings
buildInstallPSModules {} Generate .mof files for PowerShell module installation for passed nodes (prerequisite fo...
buildConfig {} Generates sample config
build {installModules, buildDSCConfig, buildInstallPSModules, buildConfig} Build project. Build will call all those other taskkks
deploy {clean, build, deployDSCConfig, deployInstallPSModules...} Deploy will push configuration to nodes. Will call build before.
deployInstallPSModules {buildInstallPSModules, {}} Deploy only module installation. Will push configuration to nodes.…Tying in Azure DevOps
In the end, when deploying code to IIS servers, I included a call to Test-DscConfiguration -Detailed to inform me whether host configuration has drifted away.
$testdsc = Test-DscConfiguration -Detailed
if (-not $testdsc.InDesiredState) {
Write-Warning "Host configuration has drifted. Test-DscConfiguration returned False"
Write-Output $testdsc.ResourcesNotInDesiredState
}
I'm not using any automatic corrections or not yet using DevOps to automatically apply configuration when DSC configuration changes have been committed. But it's possible to do that. However there comes additional complexity - perhaps I wouldn't like to reboot all servers at once if reboot is required. Or implement rolling deployment - apply configuration to some servers, see how they behave and then others.
Conclusion
I really wish PowerShell DSC would have provided OOTB way of installing modules and requiring them after installation. Anyway, I'll have the "frame-work" ready for the next DSC project.
Otherwise it's a nice tool for configuration deployment, if you can find the modules. But there are plenty.
Moreover, I do recommend looking at Building a Continuous Integration and Continuous Deployment pipeline with DSC article - it even shows how to run tests to validate your configuration.
Looking forward to any comments on what could have been done better.
Top comments (3)
Thanks for the article. I would suggest adding Azure Automation to deployment pipeline. Modules can be uploaded to automation account and then, when configuration is applied to a server, they will be pooled from Automation account to target machine automatically. Azure automation also helps to track state of the machine if configured to "Apply and monitor" mode.
I also heard rumors around custom pool server, but I never used it. It can be helpful in fully isolated environments.
@vlariono thanks for the valuable comment. So there is a better way (if using pulling strategy)! I looked at docs.microsoft.com/en-us/azure/aut... and could read what you say:
Custom pull server is actually officially documented: Desired State Configuration Pull Service.
It talks about publishing Module and .mof files to pull server. And I could find a confirmation that PowerShell would pull modules from there:
So basically adding pull server to the recipe may really simplify DSC configuration deployment. And it would be good for another reason: No dependency on PowerShell Gallery, locally available modules. That may reduce risks of npm-like attacks where malicious versions are being posted only by 3rd party and used without reviewing. Luckily it is way easier to audit a version of PS Module than an NPM package a deep dependency tree.
A way to use one script, is to put the DSC config in a "TEXT" variable, and run it as a script block.
`Install-Module -Name 'xPSDesiredStateConfiguration' -Repository PSGallery -Force -Scope AllUsers -Verbose
Install-Module -Name 'xWebAdministration' -Repository PSGallery -Force -Scope AllUsers -Verbose
Import-Module 'xPSDesiredStateConfiguration' -Force -Verbose
Import-Module 'xWebAdministration' -Force -Verbose
$dscScript = @'
Import-Module 'xPSDesiredStateConfiguration' -Force -Verbose
Import-Module 'xWebAdministration' -Force -Verbose
Configuration CreateDirectories
{
param(
[string[]]$NodeName = 'localhost',
[string[]]$modulePath = 'C:\MyModules'
)
Node $NodeName
{
File CreateModulePathDirectory
{
Type = 'Directory'
DestinationPath = "$($modulePath)" # Replace with the value of $modulePath
Ensure = 'Present'
}
}
}
FileResourceConfiguration -NodeName 'localhost'
Start-DscConfiguration -Path .\FileResourceConfiguration -Wait -Verbose
'@
$scriptBlock = [scriptblock]::Create($dscScript)
& $scriptBlock`