This is going to be about improving deployment process from classic file copy to something more effective, automated and low friction for developers and bringing no interruptions to users. Maximizing Developer Effectiveness is actually competitive advantage after all.
Pain points
Our customer, a national school management platform was experiencing inconvenience when deploying a fairly big classic .NET Framework application to 22 Windows IIS servers:
- Deployments would be taken at night
- If a hotfix had to be issued at day with high load - it could take up to 30 minutes of complete system halt as IIS server farm compiles resources while pile of requests waits in a queue.
- The deployment is not atomic - that is, while files are being copied between v1 and v2, we would get v1.314 - while some files are updated, the others not.
- If we ever wanted a rollback - occasionally files would be locked, exacerbating already bad situation.
- The website itself consists of ~220MB of data across ~2500 number of files. And small file copy is SLOW.
The root cause of this is that .NET resources (.aspx pages, razor views) get compiled to assemblies on first request whenever they change. When IIS is constantly bombarded with requests to all possible resources, it all must be compiled at the same time on every IIS server. On the other hand, when deployed at night, delay was minor as not many requests came in, and those that did, compiling was handled fairly quickly.
ASP.NET Requests Queued is a good performance counter to monitor to identify when user browsers are just waiting for response and how quickly situation resolves after an update. A healthy value is 0 or something close to that. Here you see cumulative request queue for all IIS servers in evening, when load actually is not high. 4 minutes of interruptions:
We thought about using robocopy to copy changed files based on timestamp, but that wouldn't work for us - as there could be multiple persons that would deploy the code. When Visual Studio Publishes the project, files between machines have different timestamps. Rsync would probably help greatly, but then again - some challenge to get that on developers windows machines and windows share server. And if different hosts build files, little differences in build tools may end up with different checksums. So we didn't go down this path.
The webroot (and IIS Shared Configuration) is actually served from a single Windows share, which at least provided a low effort way to conveniently copy files to a single location and keep them, along with the IIS configuration, in sync.
That's how we lived for years and it was clear that something had to change.
Deployment strategy
We came to a decision we would like to stop depending on a file share. We already started using Azure DevOps for some other stuff. So we got an idea 💡
- Keep files on IIS local disk (Duh)
- Azure DevOps Agent would be the one compiling code.
- We would precompile views in advance. Locking wouldn't be an issue as we will copy within a new folder. 💎
- Orchestrate deployment to all IIS servers via Azure DevOps Release Pipelines
- Run all .dll files through ngen. That would eliminate additional delay when assembly is first loaded. As C# code compiled is actually MSIL - an intermediate language that CPU doesn't understand any of that. JIT compiler is the one that translates that to bytecode for the processor architecture at hand. And it does so usually Just-In-Time or On-The-Fly. Or, in our case, ngen (Native Image Generator)
- A deployment task would have to copy files within a new folder and then we would change IIS Physical path from where to serve site files. 💎
So there are some gems there that are enough to solve our pain points. Now let's see some practical ways on achieving all of this!
Implementing build
Build is the process of generating files we must copy onto IIS servers.
Before we talk about build & deployment, let's get this straight: We are using Azure DevOps online offering. However this could as well be On-Premises installation or any other CI/CD platform of your choice. We are, however, using self-hosted agents that perform the build tasks.
We are talking about solution, which consists of multiple .NET projects. Our main interest is deploying ....Web.Application
and ..Web.Application.Payments
. The others are mostly dependencies which must also be built. So, the developers used to build app within Visual Studio. When deploying web application, you had to right click on particular project and choose publish. From there, some settings could be configured like whether we want precompilation or not, whether we deploy straight to IIS or filesystem. In our case, it was deployed to filesystem and then copied to production. More about Publishing an ASP.NET web app.
When writing Azure DevOps pipeline, I have to figure out what kind of MSBuild properties I have to use to invoke the "Publish" process. So, after searching the web, reading MSBuild diagnostic output logs, reading MSBuild .target
files, I'v come up with properties I need.
By the way, here is a quick tip on how to find relevant .target files by some keyword:
> gci -Path "${env:ProgramFiles(x86)}\Microsoft Visual Studio\" -Recurse -Filter "*.targets" | sls "MvcBuildViews" -SimpleMatch -List C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\Microsoft\VisualStudio\v15.0\Web\Microsoft.Web.Publishing.targets:849: <Target Name="CleanupForBuildMvcViews" Condition=" '$(_EnableCleanOnBuildForMvcViews)'=='true' and '$(MVCBuildViews)'=='true' " BeforeTargets="MvcBuildViews"> C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VisualStudio\v16.0\Web\Microsoft.Web.Publishing.targets:847: <Target Name="CleanupForBuildMvcViews" Condition=" '$(_EnableCleanOnBuildForMvcViews)'=='true' and '$(MVCBuildViews)'=='true' " BeforeTargets="MvcBuildViews">
Or better yet, use Project System Tools extension with MSBuild Binary and Structured Log Viewer to sneak peek into what MSBuild is doing - you won't regret it.
So, the relevant "Publish" command within Azure Devops build pipeline:
- task: VSBuild@1
displayName: Publish Web.Application
inputs:
solution: 'Web.Application'
msbuildArgs: >
/t:Build,GatherAllFilesToPublish
/p:PublishProfileName="$(publishProfileName)"
/p:WebPublishMethod=FileSystem
/p:DeleteExistingFiles=true
/p:DeployOnBuild=true
/p:MvcBuildViews="${{ parameters.Precompile }}"
/p:PrecompileBeforePublish="${{ parameters.Precompile }}"
/p:WDPMergeOption="MergeAllOutputsToASingleAssembly"
/p:SingleAssemblyName="Web.Application.Precompiled"
/p:UseMerge="true"
/p:DebugSymbols="True"
/p:EnableUpdateable="False"
/p:PublishUrl="$(Build.BinariesDirectory)\my"
/p:WPPAllFilesInSingleFolder="$(Build.BinariesDirectory)\my"
/p:RunNpmScripts="$(runNpmScripts)"
/p:AutoParameterizationWebConfigConnectionStrings="false"
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
msbuildArchitecture: x64
logFileVerbosity: detailed
- Note that specifying
publishProfileName
, I use the actual publish profile used by Visual Studio Publish process and then override some properties by passing/p
msbuild arguments. - The
WebPublishMethod
ensures build files are put withinPublishUrl
folder. -
RunNpmScripts
is our own custom build property used within.pubxml
to run some npm build process. -
MvcBuildViews
,PrecompileBeforePublish
,SingleAssemblyName
,UseMerge
,EnableUpdateable
all relate to precompilation. Before build, I can choose to disable precompilation if I want the build to happen much faster. -
AutoParameterizationWebConfigConnectionStrings
- required for copy/paste if publish profile contains connection string replacements. Without this, there will be a placeholder value that must be replaced afterwards. More on StackOverflow.
The full build can be seen here:
# ASP.NET
# Build and test ASP.NET projects.
# Add steps that publish symbols, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/apps/aspnet/build-aspnet-4
trigger: none
name: $(Build.SourceBranchName) $(Date:yyyyMMdd)$(Rev:.r)
pool:
name: win-dev-pool
parameters:
- name: Precompile
default: false
type: boolean
displayName: Precompile
variables:
webApplicationProject: 'Web.Application'
solution: 'solution.sln'
buildPlatform: 'x64'
buildConfiguration: 'Release'
releaseArchiveFilename: 'webapp.7z'
releasePaymentsArchiveFilename: 'payments.7z'
# which .pubxml file to use. Don't append .pubxml
publishProfileName: STAGING
runNpmScripts: true
artifactShareName: \\MY-APP-AGENT2\agentpublishedfiles
steps:
- task: NuGetToolInstaller@1
- task: NuGetCommand@2
inputs:
command: 'restore'
restoreSolution: '$(solution)'
feedsToUse: 'config'
nugetConfigPath: 'NuGet.Config'
# npm script prerequisites
- task: NodeTool@0
inputs:
versionSpec: '14.x'
# https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#nodejsnpm
- task: Cache@2
displayName: Cache npm
inputs:
key: 'v2 | npm | "$(Agent.OS)" | $(webApplicationProject)/package.json'
path: '$(webApplicationProject)/node_modules'
restoreKeys: 'v2 | npm | "$(Agent.OS)"'
cacheHitVar: NPM_CACHE_RESTORED
condition: and(succeeded(), variables.runNpmScripts)
- task: npmAuthenticate@0
inputs:
workingFile: '$(webApplicationProject)/.npmrc'
- task: Npm@1
displayName: 'npm install'
inputs:
command: 'install'
workingDir: '$(webApplicationProject)/'
condition: and(succeeded(), variables.runNpmScripts, ne(variables.NPM_CACHE_RESTORED, 'true'))
# Workaround for AspNetPrecompile to skip scanning node_modules directory and finding .c,.cpp,.h files... https://stackoverflow.com/a/20963170/50173
- task: CmdLine@2
displayName: Hide node_modules.
inputs:
script: 'attrib +H $(webApplicationProject)/node_modules'
# Specificually pass PublishProfileName as empty. Because otherwise in consequent runs, this task will run npm run-script that is specified as BeforeBuild target within publish profile.
# We already run publish actions further.
- task: VSBuild@1
displayName: Build $(solution)
inputs:
solution: '$(solution)'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
msbuildArchitecture: x64
logFileVerbosity: detailed
msbuildArgs: >
/p:PublishProfileName=""
- task: VSTest@2
displayName: Test $(solution)
inputs:
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
condition: ne(variables.NoTests, true)
# Target also Build, otherwise BeforeBuild target won't execute. BeforeBuild specifies npm run-script commands. So, this project gets built twice, but project build by itself is fast.
# PublishUrl actually unused as we won't use msdeploy to deploy stuff, just simple copy.
# Precompilation decision matrix: https://docs.microsoft.com/en-us/previous-versions/aspnet/bb398860(v=vs.100)
- task: VSBuild@1
displayName: Publish Web.Application
inputs:
solution: 'Web.Application'
msbuildArgs: >
/t:Build,GatherAllFilesToPublish
/p:PublishProfileName="$(publishProfileName)"
/p:WebPublishMethod=FileSystem
/p:DeleteExistingFiles=true
/p:DeployOnBuild=true
/p:MvcBuildViews="${{ parameters.Precompile }}"
/p:PrecompileBeforePublish="${{ parameters.Precompile }}"
/p:WDPMergeOption="MergeAllOutputsToASingleAssembly"
/p:SingleAssemblyName="Web.Application.Precompiled"
/p:UseMerge="true"
/p:DebugSymbols="True"
/p:EnableUpdateable="False"
/p:PublishUrl="$(Build.BinariesDirectory)\my"
/p:WPPAllFilesInSingleFolder="$(Build.BinariesDirectory)\my"
/p:RunNpmScripts="$(runNpmScripts)"
/p:AutoParameterizationWebConfigConnectionStrings="false"
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
msbuildArchitecture: x64
logFileVerbosity: detailed
- task: VSBuild@1
displayName: Publish Web.Application.Payments
inputs:
solution: 'Web.Application.Payments'
msbuildArgs: >
/t:Build,GatherAllFilesToPublish
/p:PublishProfileName="$(publishProfileName)"
/p:WebPublishMethod=FileSystem
/p:DeleteExistingFiles=true
/p:DeployOnBuild=true
/p:MvcBuildViews="${{ parameters.Precompile }}"
/p:PrecompileBeforePublish="${{ parameters.Precompile }}"
/p:WDPMergeOption="MergeAllOutputsToASingleAssembly"
/p:SingleAssemblyName="Web.Application.Payments.Precompiled"
/p:UseMerge="true"
/p:DebugSymbols="True"
/p:EnableUpdateable="False"
/p:PublishUrl="$(Build.BinariesDirectory)\payments"
/p:WPPAllFilesInSingleFolder="$(Build.BinariesDirectory)\payments"
/p:AutoParameterizationWebConfigConnectionStrings="false"
platform: 'AnyCPU'
configuration: '$(buildConfiguration)'
msbuildArchitecture: x64
logFileVerbosity: detailed
# Artifacts
- task: ArchiveFiles@2
displayName: Archive $(releaseArchiveFilename)
inputs:
rootFolderOrFile: '$(Build.BinariesDirectory)\my'
includeRootFolder: false
archiveType: '7z'
sevenZipCompression: 'fastest'
archiveFile: '$(Build.ArtifactStagingDirectory)/$(releaseArchiveFilename)'
replaceExistingArchive: true
verbose: true
- task: ArchiveFiles@2
displayName: Archive $(releasePaymentsArchiveFilename)
inputs:
rootFolderOrFile: '$(Build.BinariesDirectory)\payments'
includeRootFolder: false
archiveType: '7z'
sevenZipCompression: 'fastest'
archiveFile: '$(Build.ArtifactStagingDirectory)/$(releasePaymentsArchiveFilename)'
replaceExistingArchive: true
verbose: true
- task: PublishBuildArtifacts@1
displayName: Publish website deployment Artifacts
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'FilePath'
TargetPath: '$(artifactShareName)\my'
- What this does, is builds solution (dependency .dlls)
- Builds JavaScript SPA (Single Page App). Commands are buried within MSBuild project.
- Publishes 2 applications (Just generates files on disk)
- 7zips those files
- Publish artifact (Azure Devops thingie - makes them available for release pipeline). In this case, they are copied to a share.
Implementing deployment
Once we have the artifacts ready (archive of website files), we are ready to implement Release. The release must copy given files to some directory and instruct IIS to change base path from where it will server files. We had 2 options to choose from:
- Make Build agent connect to IIS servers and perform the deployment on each IIS server. In this case, we have to write some script that will copy files onto each server and issue some commands to IIS. Moreover, we must control/see whether deployment is successful or not. And the build agent could actually be in another domain, with no direct access to production.
- Install Deployment agent on all target IIS servers. Every host then receives deployment job and does whatever it is instructed to do. We don't have to bother about any other remote communication channel other than Deployment Agent with Azure DevOps server over 443/TCP. Plus we get nice UI of seeing whether deployment succeeded, partially succeeded (if partially, which hosts failed) and on which step it failed. Neat.
We went for the second option. Deployment agent is actually almost same as Build agent, just carries on deployment tasks.
When creating release pipeline, you get to play with the UI which is nice.
As you see, I'v split some operations into multiple steps.
- Root App Pool - ensures appropriate application pool is created on IIS. It is actually a one-time step and may have been created beforehand.
- Copy my files - PowerShell script to extract 7z files:
Expand-7Zip -ArchiveFileName "$(System.DefaultWorkingDirectory)\_Web.Application\drop\webapp.7z" -TargetPath "$(DeployTargetFolder)"
-
Ngen my - runs
ngen.exe
on all .dll files found within deployment folder. Also using custom condition so this step which may take a little more than a minute, could be turned off:and(succeeded(), eq(variables.Ngen, 'true'))
Set-Alias ngen -Value (Get-ChildItem -Recurse $env:windir\Microsoft.NET\Framework64\ -Filter ngen.exe | ? Length -gt 0 | select -first 1).Fullname
Get-ChildItem $(DeployTargetFolder) -Recurse -Filter *.dll | % { ngen install $_.Fullname /nologo /verbose }
- Deploy my - Switchers virtual directory, issues some IIS configuration commands. When this stage is completed, IIS servers new code.
The great thing is that if any of steps fail for ANY IIS server before Deploy step, deploy won't run for ANY IIS server and they will all be still consistent.
In case of a rollback, we can open appropriate release and for "Deploy my" stage, press Redeploy. IIS will immediately switch back to appropriate folder.
Drawbacks
Using cloud hosted solution, if issues do happen, we won't be able to run our deployment pipeline. And they do happen. However, this may impact us on the rare case of rushing some kind of hotfix out. There are 2 steps we can take in this case:
- Change IIS path manually to previous deployment folder (rollback)
- Build via developer Visual Studio and copy appropriate DLL manually. Hotfix usually doesn't involve much code changes and is probably contained within a single or few files.
Otherwise we just wait for while DevOps issue gets resolved.
End result
Everything is actually configured that, when production
branch gets new code, build is run automatically. After build, deployment is run automatically but stops for an approval. With Azure Pipelines Slack app we just get a message within channel where upon Approve button press, code goes into production.
This doesn't include database updates, which still must be performed manually, but eventually DACPAC deployment can be incorporated within a pipeline too. But luckily, database updates are more rare than backend code/frontend updates.
Judging from the Request Queue size, guess where did deployment happen?
Yeah, that little spike just before 09:00. Except, vertical axis shows max 30 instead of 3k and horizontally, just a blip of a time.
For a fair picture I should share another deployment request queue graphic:
Queue size spiked to almost 300. However it happens on some IIS servers, in this case 3 out of 10. After 60-90 seconds, queue size dropped, so fairly short period of time compared to what we experienced before. But it's not like those IIS completely stopped processing GET/POST requests - actually only minority of requests queued up. Currently I don't know the cause. Maybe if you have any thoughts on what may cause it, leave down in the comments.
In the end, the results leave everyone happy!
On an upcoming post, I'd like to share how these IIS server/Windows OS settings can be installed, managed and kept in sync with PowerShell DSC. And you don't have to keep a separate documentation file somewhere that may drift and be outdated in time. Stay tuned! 👋
Top comments (0)