DEV Community: Anton Kuryan

Azure Web app Kudu: overcoming api/zip built-in timeout

Anton Kuryan — Wed, 15 Sep 2021 11:55:49 +0000

Azure Web apps comes in with very interesting tool, called Kudu, which is web application itself, designed to service and configure your web application. It exposes several APIs to service you - for example, MsDeploy, zip, zipdeploy. In this post I wish to cover some differences between zip and zipdeploy and how I used zip API to deploy Sitecore Unicorn data.

First of all, I wish to describe my problem - we have huge Sitecore 9 web application, which required to deploy 18000 Unicorn files to CM instance in scope of every release outside of web root to solve issues with long paths (we were deploying at C:\home\N directory).

Initially, I tried to solve this problem with MsDeploy - it seems to be fast and reliable and it is designed to deploy you web application to remote server. It worked, but it took too much time due to retries in-between. As I discovered, deployment of 18000 tiny files would not be the best things MsDeploy is able to handle reliably. So, I started looking in other possibilities and discovered, that Kudu exposes 2 more APIs which could be used for deployment: Zip and ZipDeploy.

Zip vs ZipDeploy

ZipDeploy have following possibilities

Delivers files ONLY to webroot (there is some magic which one could do with NPM, but I never finished this exercise)
Support both sync and async behaviors (async could be required to overcome built in app services hard-coded timeout of 230 seconds per request)
Smart as MsDeploy - would not update files that have not changed; would remove files which is not present in archive

Zip have following possibilities

Delivers files almost to any place of web app work drive
Does not remove files which is not present in archive
Have only sync behavior
Continues execution in background even after request have timed out

Following up on my explanation where I required to deploy 18000 tiny yml files for Sitecore Unicorn outside of webroot - I have been left with only option: using zip API. But, most of my tries it required more than 230 seconds to deliver files on web app, which means that I was not able to reliably find out if unzip operation have been finished.

Solution

I implemented following solution - in each archive with Unicorn files I am adding to the root of archive some unique text file (I am using build number as name and content). From my observations, this file would be extracted last. So, I am feeding api/zip with my Unicorn archive and then, if it fails with timeout, I will just keep asking api/vfs (virtual file system provider) for this file. When file appears there - it means that api/zip have finished it's work and I could proceed with Unicorn import routines. I shall note, that before starting deployment I have to clean up Unicorn folder, as zip API does not remove excessive files.

This small powershell script allowed me to solve my issue with deployment

SonarQube Community edition and PR analysis

Anton Kuryan — Tue, 08 Dec 2020 14:51:05 +0000

SonarQube is awesome tool and it's Community edition (which is free) fulfills almost everything one could dream for in the world of static code analysis. But, you could not analyze PR's with it by default and have to pay for at least Developer edition. Further you could find my attempt to overcome this limitation (it fulfills MY goals and seems to be worth sharing with broader audience).

When I started working with SonarQube back in version 5 and 6 there was one mode which brought me a lot of interesting discoveries and prevented my colleague developers from submitting bad code - preview. In this mode, SonarQube does not stores data on server, but executes analysis and sends results back - and in conjunction with PR decoration plugin it was very useful. But, time goes by and version 7 removed preview mode from Community edition.

So, I tried to mimic this, using a separate project on SonarQube server and created powershell script, which will retrieve quality gate status from current analysis and create bugs in Jira and attach them to some existing issue (actually, that's the added value of this script, which was my target, opposing to default CI server SonarQube runners, which would just fail your build).

Script seems to be heavily commented and speaks by itself. Feel free to adopt it for your own usage pattern, if you will ever need it.

I shall note that Developers edition of SonarQube solves the same problem much better - it have PR analysis and built-in PR decorator, but it costs money, while Community edition brings things in for free.

Transforming deployed configuration file in Azure web app

Anton Kuryan — Mon, 21 Oct 2019 07:03:35 +0000

While this guide is primarily targeted towards usage with Azure Web apps and Sitecore ASP.NET solution deployed on them, still it is possible to use it on on-premise solution with other types on ASP.NET projects, running on Windows.

Reasoning behind this

Why I actually started building all this thing up, ignoring interesting Helix publishing pipeline or dropping idea of storing source and transformation files in my source control?

In Helix publishing pipeline I do not like the fact that I could not use different transformation based on roles, where I am deploying (if it is there - can you point it to me?).

When storing vanilla config in source control - I do not like the fact that I will need to update it when I am updating Sitecore version (hence, merging and solving possible conflicts).

Solution

So, I decided to build this transformation thing anew, almost from scratch. I used the same idea as used in Helix publishing pipeline: store only transformation files under source control, as, in general, they are generic. Also, I agreed on following naming convention:

configFileName.config.xdt - transforms config file with name configFileName.config lying at this path for all environments
configFileName.config.xdt.roleName - transforms config file with name configFileName.config lying at this path when deploying this particular role

That's is a script, which could be uploaded to web app and executed via Kudu. Script wants several parameters:

folderWithTransforms - (optional) folder, where all transform files are residing; if not specified - script folder is taken
webRoot - (optional) folder, where webRoot could be found; if not specified, folder 2 levels upper is taken
roleName - optional, if there is a specific tranformations for this particular role
transformationAssemblyPath - if Microsoft.Web.XmlTransform.dll is not at the same path as script

Road to precompiled web application based on Umbraco CMS

Anton Kuryan — Mon, 19 Aug 2019 09:25:33 +0000

This is crosspost from my blog post.

Starting the journey

Some time ago, my colleague Jeroen asked me to optimize the local development process for one of our web applications, which has more than 450 C# MVC views, which leads to a problem – extremely long local startup times (can be more than 5 minutes on decent developer workstation). Also, we’ve experienced the same problems on production, which is delivered by means of MsDeploy (though they are not so big, cause delivery to production is much rarer event than local compilation). So, I set sails on investigation and optimizing this

First try: MvcBuildViews

In each C# MVC project there is an option to compile MVC views, by turning parameter MvcBuildViews to True. I supposed that this will solve our issue by compiling views to IL (Intermediate Language), but this was not the case. As I understand, the main target of this parameter is to check views for errors, so we can avoid them during runtime compilation in future – it starts aspnet_compiler.exe to compile all views to IL (this is what seems to be the goal), but, it keeps views itself unmodified and compiles to a temporary folder. So, on the next startup of web application, when a view is requested, it will be loaded unmodified from the Views folder and undergo the whole runtime compilation again, outputting IL to another temporary folder. Additional catch is: MvcBuildViews is not compatible with MsDeploy, so, if you are compiling and deploying in one run (e.g. parameter DeployOnBuild is set to True) – the build will fail, as ASPCONFIG is failing on parsing application web.config. which fails a build. However, if you do wish to have your views to be checked for compilation sanity during deployment builds, this can be fixed by modifying build process (for example, in csproj file of your MVC application) by adding the following at the end (before closing tag):

<PropertyGroup>
<!-- MvcBuildViews is not compatible with DeployOnBuild - so we need to change order of build events -->
<MvcBuildViews Condition="$(MvcBuildViews) == ''">False</MvcBuildViews>
<MSDeployPublishDependsOn Condition="$(DeployOnBuild) And '$(MSDeployPublishDependsOn)'!=''">MvcBuildViews;$(MSDeployPublishDependsOn);</MSDeployPublishDependsOn>
</PropertyGroup>
<Target Name="MvcBuildViews" Condition="$(MvcBuildViews)">
<AspNetCompiler VirtualPath="temp" PhysicalPath="$(WebProjectOutputDir)" />
</Target>

This will modify the build process to ensure that MvcBuildViews is performed before publishing (by default, it is performed after publish step) and will allow to drop an error from compiler about web.config. However, it will still not produce precompiled IL for views and the application must build it on runtime.

Second try: Built-in views precompilation

It turns out that the Visual Studio team already thought about those, who wish to use precompiled views and embedded set of parameters to be set to achieve this goal, which is invoked on publish of web application. So, it pushes us to using out-of-webroot development, which is beneficial for all (allows to tackle edge cases, encountered during publishing to production; makes your solution smaller and isolates delta of your custom work on top of the vanilla install), because view precompilation alters *.cshtml files (they are left there as a placeholders, with placeholder text).

So, I used the publishing wizard and configured it as follows:

If one will set ‘Allow precompiled site to be updateable’ – precompilation will not fire up for Views (at least, it does not produce AppCode.dll in bin folder of published app). This ended up in the following publish profile (all parameters there can be copied directly in csproj file of your MVC web app):

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<WebPublishMethod>FileSystem</WebPublishMethod>
<LastUsedBuildConfiguration>Debug</LastUsedBuildConfiguration>
<LastUsedPlatform>Any CPU</LastUsedPlatform>
<SiteUrlToLaunchAfterPublish />
<LaunchSiteAfterPublish>False</LaunchSiteAfterPublish>
<ExcludeApp_Data>False</ExcludeApp_Data>
<publishUrl>../../Published</publishUrl>
<DeleteExistingFiles>False</DeleteExistingFiles>
<PrecompileBeforePublish>True</PrecompileBeforePublish>
<EnableUpdateable>False</EnableUpdateable>
<DebugSymbols>True</DebugSymbols>
<WDPMergeOption>MergeAllOutputsToASingleAssembly</WDPMergeOption>
<UseMerge>True</UseMerge>
<SingleAssemblyName>AppCode</SingleAssemblyName>
<DeleteAppCodeCompiledFiles>False</DeleteAppCodeCompiledFiles>
</PropertyGroup>
</Project>

Then I tried to publish my web app and … it fails.

Catch 1: Umbraco and Umbraco_client folders

Due to some historical reasons, we’ve included both umbraco and umbraco_client in the solution, which lead to a precompilation failure. There are several ascx files, which are referring to backend code which is not present anymore. However, if Umbraco is installed via a NuGet package – it is not needed to include umbraco and umbraco_client folders in the solution, because nuget will import a set of tasks, which is responsible for copying umbraco and umbraco_client content to web deploy package and solution root during build. So, I excluded those folders again, and … it failed again. This time it failed because we have had some plugins installed and included in our solution, which was using masters from umbraco folders. I Included those masters files – and this time, precompilation was OK.

Catch 2: App_Code folder

I expected that after publishing my app will start up fast and will not spend any time on runtime precompilation, but – I received YSOD, stating that App_Code folder is not allowed in precompiled applications. I checked filesystem and found out that App_code with some cs files is present there. It turned out that we have had some cs files, marked as content in this folder and used by nuPicker. I changed their attribute to ‘Compile’ and published once again (clearing target folder manually, as my publish profile says to keep existing files by setting DeleteExistingFiles to False – this is needed as we have a separate frontend build, which uses Gulp for generating it). After that, application shown up and startup time improved greatly (though, initial index build requires some time, but subsequent publishes result in fast application start).

Catch 3: Long publish times

Views precompilation is good, but there is one immediately observed tradeoff – time to build increases greatly (on HDD can be up to 5 minutes – the same as we spent on startup before precompilation). So, in order to benefit from fast startup, my colleague Jeroen, who initially came with the idea of this feature, added a simple postbuild event in our MVC application csproj file:

<PropertyGroup>
<PostBuildEvent>XCOPY "$(ProjectDir)bin\*.dll" "$(ProjectDir)..\..\Published\bin\" /S /Y /i</PostBuildEvent>
</PropertyGroup>

As you can see – after build it will copy all dll’s from project bin to published folder, where precompiled app resides. So, if one has not changed anything in views – he does not need to launch full publish, but just build the solution.

Catch 4: Could not login to Umbraco

After playing enough with frontend, we’ve found out that trying to login to Umbraco results in YSOD. Quick check revealed its source – since our application is precompiled and not updateable (see publish profile example) it is expecting that all parts of our application are precompiled in the same manner. But, as noted before, we could not precompile Umbraco and umbraco_client folders, as this leads to failures in precompilation stage. Looks like a real problem, but we’ve found 2 possible solutions:

1) We can trick runtime: when app is precompiled, compiler will drop PrecompiledApp.config in webroot and add a definition there, if app is updateable or not: So, to allow using non-precompiled Umbraco views with precompiled web app views – just set updateable to true This can be done by adding the following to MVC app csproj file

<Target Name="__localPublishAllowUpdateable" AfterTargets="CopyAllFilesToSingleFolderForPackage" Condition="$(PrecompileBeforePublish) And !$(EnableUpdateable)">
<!-- This target is used with local publish -->
<MSBuild Projects="$(MSBuildProjectFile)" Targets="__AllowUpdateable" />
</Target>
<Target Name="__MsDeployPublishAllowUpdateable" AfterTargets="CopyAllFilesToSingleFolderForMsdeploy" Condition="$(PrecompileBeforePublish) And !$(EnableUpdateable)">
<!-- This target is used with msdeploy publish -->
<MSBuild Projects="$(MSBuildProjectFile)" Targets="__AllowUpdateable" />
</Target>
<Target Name="__AllowUpdateable" Condition="$(PrecompileBeforePublish) And !$(EnableUpdateable)">
<!-- This target is required to mark precompiled app, based on Umbraco CMS as updateable -->
<PropertyGroup>
<___IntermediateOutputPath Condition="'$([System.IO.Path]::IsPathRooted($(IntermediateOutputPath)))' == 'False'">$(MSBuildProjectDirectory)\$(IntermediateOutputPath)</___IntermediateOutputPath>
<___IntermediateOutputPath Condition="'$([System.IO.Path]::IsPathRooted($(IntermediateOutputPath)))' == 'True'">$(IntermediateOutputPath)</___IntermediateOutputPath>
</PropertyGroup>
<Exec Command='copy "$(ProjectDir)Properties\BuildTargets\PrecompiledApp.source" "$(___IntermediateOutputPath)Package\PackageTmp\PrecompiledApp.config" /y' />
<RemoveDir Directories="$(___IntermediateOutputPath)Package\PackageTmp\App_Code" />
</Target>

These targets will copy PrecompiledApp.source file from Properties\BuildTargets of MVC web app project to output folder of published app. Content of PrecompiledApp.source is:

<precompiledApp version="2" updatable="true"/>

Actually, this can be achieved by importing this nuget - http://www.nuget.org/packages/Colours.Ci.Umbraco/

2) Another possible solution, which also addresses catch number 5 and can be used to test load balanced solution is to add 2 IIS sites: one is pointing to published folder, and we will use it to check web app performance, check changes in code and views to be displayed correctly, second – pointing to folder with MVC project, which we will use to access Umbraco. In such approach, besides all other benefits, we can easily test load balanced approach (changing content in Umbraco shall be reflected in our published precompiled web app, events shall be called on both web apps and etc).

Catch 5: Could not debug views

Despite the fact, that we are publishing in Debug mode and are adding DebugSymbols (see publishing profile) – views could not be debugged when debugger is attached to published web app process. Hence, Jeroen came up with a solution, described as point 2 in Catch 4. At this moment, I could not figure out how to debug views in precompiled application – maybe someone can come up with suggestions?

Catch 6: VPP is not working

Our Umbraco based web app has the following setup: it is storing images at Azure blob storage via UmbracoFileSystemProviders.Azure, processes them via ImageProcessor and serves them via Azure CDN with the help of Our.Umbraco.AzureCDNToolkit. Final problem we faced was – our web app was not able to serve media directly from our virtual path provider (VPP), without processing it via ImageProcessor. So, all request like http://mydevhost/media/1000/1.jpg was serving ASP.NET 404, while requests like http://mydevhost/media/1000/1.jpg?1=1 will be happily picked up by ImageProcessor and processed further. Not a big flaw, and, in some setups, it can be even treated like an additional source of protection for original images (imagine that you do not want to serve original media items for your end-users in any circumstances). Actually, this is by design: http://msdn2.microsoft.com/en-us/library/system.web.hosting.virtualpathprovider.aspx notes that VPP will not work in precompiled web application.
However, in this forum thread I came up with an answer how we can hack it and allow usage of particular VPP in precompiled application.

Teamcity configuration

Since we are using direct publish from Teamcity (system.DeployOnBuild is set to True) via MsDeployPublish – we have 2 options to set this up:

Define all parameters in additional publish profile and pass its name to Teamcity build
Add following parameters as system to the build configuration itself:

system.EnableUpdateable = False
system.PrecompileBeforePublish = True
system.SingleAssemblyName = nameOfTheAssembly
system.UseMerge = True

However, in my situation, despite of the fact that I have had Visual Studio 2015 and SDK’s for .NET versions installed – build was failing, because Teamcity was not able to find aspnet_merge.exe, which is part of the SDK. So, I defined 2 additional parameters:

system.AspnetMergePath = “C:\Program Files (x86)\Microsoft SDKs\Windows\v8.1A\bin\NETFX 4.5.1 Tools\”
system.GetAspNetMergePath = False

After this, my builds were green again and I got a precompiled application also on our production environment, which is boasting faster startup times after IIS recycle or VM restart for updates installation.

Automated umbracoSettings.config modifications

Anton Kuryan — Mon, 19 Aug 2019 08:21:11 +0000

This is cross-post from my blog.

This is one more piece of automation puzzle to save time and make the life of a build engineer easier. This part is targeted towards removing double work of redoing things that are already done on Teamcity level configurations of Umbraco-driven Continuous Delivery (CD) projects.

Umbraco is an open-source ASP.NET CMS with a big developers community. One of the features of this CMS, which makes automated deliveries somewhat harder to configure for build engineer, is a configuration file system, which consists out of web.config and bunch of other .config files, stored in the ~/config directory. This is brilliant idea - each Umbraco part is driven by its own configuration file, but it becomes a nightmare when you have to setup some environment-specific settings in some of these files. Configuration file transformation, even in Visual Studio 2015, allows you to build web.config file transforms only. Of course, there is the SlowCheetah plugin, which alleviates this problem, but it generates extra work for the build engineer. Since we have an established automated Continuous Delivery process, our main target is to remove all those small distractive pieces of configuration and automate everything J.

In this post I will cover automated modifications on one of the two parts of the umbracoSettings.config file, responsible for KeepAliver, task scheduling and delayed publishing in Umbraco.

To my great grieve, Umbraco has two of such parameters, which are version dependent:

for versions 6.2.5 and 7.1.9-7.2.7 it is the baseUrl attribute of the scheduledTasks element (since 7.2.7 it is obsolete, in previous versions it is not present)
since 7.2.7 it is the umbracoApplicationUrl attribute of the web.routing element

Target of both: remove automated guess from first request about Umbraco backoffice access URL and set it up by configuration (fully, they are covered at https://our.umbraco.org/documentation/reference/config/umbracosettings/ and http://issues.umbraco.org/issue/U4-6788).

Code and explanations

First of all, we have to modify the MsBuild compilation process to make sure that this task is executed directly after compilation and before configuration files transformation, by adding it to the centralized project file PropertyGroup element:

<PropertyGroup>
    <TargetsTriggeredByCompilation>
        $(TargetsTriggeredByCompilation);
        UmbracoSettingsConfigTransform; 
    </TargetsTriggeredByCompilation>    
</PropertyGroup>

This will ensure that our target UmbracoSettingsConfigTransform is called directly after compilation. Next to it, as flexibility is required, in the same PropertyGroup we add some variables, which allows to stop automated transformation:

<PropertyGroup>
    <!-- If system.DoNotSetScheduledTasksBaseUrl is not defined - it shall be equal to false -->
    <DoNotSetScheduledTasksBaseUrl Condition="'$(DoNotSetScheduledTasksBaseUrl)'==''">False</DoNotSetScheduledTasksBaseUrl>
    <!--
As seen by name starting with underscores - __SetScheduledTasksBaseUrl - is internal variable.
It's target - to establish setup of scheduled task base url
-->
    <__SetScheduledTasksBaseUrl>!$(DoNotSetScheduledTasksBaseUrl)</__SetScheduledTasksBaseUrl>
    <!-- If system.DoNotSetUmbracoApplicationUrl is not defined - it shall be equal to false -->
    <DoNotSetUmbracoApplicationUrl Condition="'$(DoNotSetUmbracoApplicationUrl)'==''">False</DoNotSetUmbracoApplicationUrl>
    <__SetUmbracoApplicationUrl>!$(DoNotSetUmbracoApplicationUrl)</__SetUmbracoApplicationUrl>
</PropertyGroup>

As one can see, if the variables DoNotSetScheduledTasksBaseUrl and DoNotSetUmbracoApplicationUrl are not set, they are made False, thus allowing further transforms. If variables are set, the build process will receive the corresponding values. After that, the build process will copy the original, unmodified version to temporary storage. After the build is finished, the unmodified version will be placed back to alleviate possible version control system (VCS) checkout issues:

<Target Name="CopySourceTransformFiles" BeforeTargets="Cleanup" >
    <Copy SourceFiles="$(UmbracoSettingsConfigFullPath)" DestinationFolder="$(IntermediateOutputPath)" OverwriteReadOnlyFiles="True" />
</Target>
<Target Name="RestoreSourceTransformFiles" BeforeTargets="Cleanup" >
    <Copy SourceFiles="$(IntermediateOutputPath)$(UmbracoSettingsConfigName)" DestinationFolder="$(umbracoConfigFolder)" Condition="$(__SetScheduledTasksBaseUrl) Or $(__SetUmbracoApplicationUrl)" />
</Target>

Next to it, we have to check, if one of 2 possible modifications is valid for the current Umbraco version. Umbraco stores the currently installed version in web.config, in the appSettings section, in the value of a key umbracoConfigurationStatus. So, on build, code is retrieving this value using XmlPeek MsBuild task, then, using Nuget package SemanticVersioning (https://www.nuget.org/packages/SemanticVersioning/Открывается в новом окне,https://github.com/adamreeve/semver.netОткрывается в новом окне) version is compared against the range using custom MsBuild tasks and the internal variable receives True or False value (to allow or disallow modification):

<UsingTask TaskName="CompareSemanticVersions" AssemblyFile="$(TeamCityCiToolsPath)\CI.Builds\MsBuildCustomTasks\Colours.Ci.MSBuild.CustomTasks.dll"/>
  <Target Name="EstablishConfigFileModificationsRequirements" Condition="$(__umbraco_config_setUmbracoUrl)">
    <!-- Get version from web.config -->
    <XmlPeek Namespaces="&lt;Namespace Prefix='msb' Uri='http://schemas.microsoft.com/developer/msbuild/2003'/&gt;"
          XmlInputPath="web.config"
          Query="configuration/appSettings/add[@key = 'umbracoConfigurationStatus']/@value"
          Condition="Exists('web.config')">
      <Output TaskParameter="Result" PropertyName="__umbracoVersionFromWebConfig" />
    </XmlPeek>
    <Message Text="Umbraco version peeked from web.config: $(__umbracoVersionFromWebConfig)" Importance="high" />
    <!--
            Now we have to check, which Umbraco version we are working with.
            baseUrl for scheduled tasks are allowed in 6.2.5 and 7.1.9 - 7.2.7
            If version does not fit in this range - baseUrl shall not be set.
            If version is 7.2.7 and higher - web.routing attribute umbracoApplicationUrl shall be used
        -->
    <CompareSemanticVersions CurrentVersion="$(__umbracoVersionFromWebConfig)" AllowedVersionRange="6.2.5 || &gt;=7.1.9 &lt;7.2.7">
      <Output TaskParameter="VersionIsInRange" PropertyName="__SetScheduledTasksBaseUrl"/>
    </CompareSemanticVersions>
    <Message Text="We are going to set scheduledTasks baseUrl" Importance="high" Condition="$(__SetScheduledTasksBaseUrl)"/>
    <Message Text="We are NOT going to set scheduledTasks baseUrl" Importance="high" Condition="!$(__SetScheduledTasksBaseUrl)"/>
    <MSBuild Projects="$(MSBuildProjectFile)" Properties="CustomConfigFileToTransfom=$(_PackageTempDir)\$(UmbracoSettingsConfigFullPath);CustomConfigTransformFile=$(TeamCityCiToolsPath)\CI.Builds\SharedFiles\Umbraco\umbraco.umbracosettings.scheduledurl.config.transform" Targets="TransformCustomConfigFile" Condition="$(__SetScheduledTasksBaseUrl) AND Exists('$(_PackageTempDir)\$(UmbracoSettingsConfigFullPath)')" />
    <CompareSemanticVersions CurrentVersion="$(__umbracoVersionFromWebConfig)" AllowedVersionRange="&gt;=7.2.7" >
      <Output TaskParameter="VersionIsInRange" PropertyName="__SetUmbracoApplicationUrl"/>
    </CompareSemanticVersions>
    <Message Text="We are going to set web.routing umbracoApplicationUrl" Importance="high" Condition="$(__SetUmbracoApplicationUrl)"/>
    <Message Text="We are NOT going to set web.routing umbracoApplicationUrl" Importance="high" Condition="!$(__SetUmbracoApplicationUrl)"/>
    <MSBuild Projects="$(MSBuildProjectFile)" Properties="CustomConfigFileToTransfom=$(_PackageTempDir)\$(UmbracoSettingsConfigFullPath);CustomConfigTransformFile=$(TeamCityCiToolsPath)\CI.Builds\SharedFiles\Umbraco\umbraco.umbracosettings.umbracoApplicationUrl.config.transform" Targets="TransformCustomConfigFile" Condition="$(__SetUmbracoApplicationUrl) AND Exists('$(_PackageTempDir)\$(UmbracoSettingsConfigFullPath)')" />
  </Target>

I wish to focus on custom task:

<CompareSemanticVersions
CurrentVersion="$(__umbracoVersionFromWebConfig)"
AllowedVersionRange="6.2.5 || &gt;=7.1.9 &lt;7.2.7"
Condition="$(__SetScheduledTasksBaseUrl)">
<Output TaskParameter="VersionIsInRange" PropertyName="__SetScheduledTasksBaseUrl"/>
</CompareSemanticVersions>

Code of this task is simple:

namespace Colours.Ci.MSBuild.CustomTasks.HelperMethods.StringMethods
{
    using System;

    using Loging;

    using Microsoft.Build.Framework;
    using Microsoft.Build.Utilities;

    using SemVer;

    using Version = SemVer.Version;

    public class CompareSemanticVersions : Task
    {
        /// <summary>
        /// Current version of software
        /// </summary>
        [Required]
        public string CurrentVersion { get; set; }
        /// <summary>
        /// Allowed versions range of software.
        /// Shall use ranges, as described https://github.com/npm/node-semver#ranges
        /// Also, see https://github.com/adamreeve/semver.net#ranges
        /// </summary>
        [Required]
        public string AllowedVersionRange { get; set; }
        /// <summary>
        /// Demonstrates, if version is in range
        /// </summary>
        [Output]
        public bool VersionIsInRange { get; private set; }

        public override bool Execute()
        {
            VersionIsInRange = CheckVersions(CurrentVersion, AllowedVersionRange);
            return true;
        }

        public static bool CheckVersions(string currentVersion, string allowedVersionRange)
        {
            var logWriter = new Logging();
            Version version;
            try
            {
                version = new Version(currentVersion);
            }
            catch (Exception exception)
            {
                logWriter.LogWriter(string.Concat("There was an error ", exception.Message, "while parsing ", currentVersion));
                return false;
            }

            var range = new Range(allowedVersionRange);
            return range.IsSatisfied(version);
        }
    }
}

Actually, it is just a code, compiled to assembly. It receives the current version in CurrentVersion parameter, a range of allowed versions in AllowedVersionsRange parameters in NPM notation (https://github.com/npm/node-semver#ranges) and outputs Boolean variable. The same task is called to check version requirements for the umbracoApplicationUrl parameter. There is one MsBuild feature which shall be known - characters ‘<’ and ‘>’ must be escaped. The symbol ‘<’ is represented as ‘<’. The symbol ‘>’ is represented as ‘>’. After checking versions, we have to modify umbracoSettings.config by using the XmlTransform task, as (surprise!!!) XmlPoke tasks, used to add URL to attribute could not insert missing attributes. So, it was added static transform files, which just adds attributes with an ‘empty’ value:

<MSBuild
Projects="$(MSBuildProjectFile)"
Properties="CustomConfigFileToTransfom=$(UmbracoSettingsConfigFullPath);CustomConfigTransformFile=$(TeamCityCiToolsPath)\CI.Builds\SharedFiles\Umbraco\umbraco.umbracosettings.umbracoApplicationUrl.config.transform"
Targets="TransformCustomConfigFile"
Condition="$(__SetUmbracoApplicationUrl)" />
<Target Name="TransformCustomConfigFile" Condition="Exists('$(CustomConfigFileToTransfom)') And Exists('$(CustomConfigTransformFile)')">
<RandomNameTask>
<Output TaskParameter="RandomName" PropertyName="RandomName" />
</RandomNameTask>
<!-- Somebody have to ensure, that he have original file copied - but this shall not be done here, IMHO -->
<!-- let us do a transformation -->
<MakeDir Directories="$(IntermediateOutputPath)$(RandomName)\"/>
<TransformXml
Source="$(CustomConfigFileToTransfom)"
Destination="$(IntermediateOutputPath)$(RandomName)\transformed.$(RandomName).config"
Transform="$(CustomConfigTransformFile)"
StackTrace="True" />
<!-- now, result of transformation shall be copied to original place -->
<Copy
SourceFiles="$(IntermediateOutputPath)$(RandomName)\transformed.$(RandomName).config"
DestinationFiles="$(CustomConfigFileToTransfom)"
OverwriteReadOnlyFiles="True" />
<!-- And intermediate results shall be deleted -->
<Delete Files="$(IntermediateOutputPath)$(RandomName)\transformed.$(RandomName).config" />
</Target>

CustomTransformFile target:

  <!--
        Next target can be used to transform any input xml file by using transform file.
        You can invoke this target by following:
        <MSBuild
        Projects="$(MSBuildProjectFile)"
        Properties="CustomConfigFileToTransfom=$(TeamCityCiToolsPath)\CI.Builds\SharedFiles\Umbraco\umbraco.web.config;CustomConfigTransformFile=$(TeamCityCiToolsPath)\CI.Builds\SharedFiles\Umbraco\umbraco.web.config.transform"
        Targets="TransformCustomConfigFile" />

        Idea is that you a passing source config file and transform for it here
    -->
  <Target Name="TransformCustomConfigFile" Condition="Exists('$(CustomConfigFileToTransfom)') And Exists('$(CustomConfigTransformFile)')">
    <Message Text="************************************************************************" />
    <Message Text="TransformCustomConfigFile is runing :)" Importance="high" />
    <Message Text="Transform file is $(CustomConfigTransformFile)" Importance="high" />
    <Message Text="Transformimg following file - $(CustomConfigFileToTransfom)" Importance="high" />
    <Message Text="************************************************************************" />
    <RandomNameTask>
      <Output TaskParameter="RandomName" PropertyName="RandomName" />
    </RandomNameTask>
    <!-- Somebody have to ensure, that he have original file copied - but this shall not be done here, IMHO -->
    <!-- let us do a transformation -->
    <MakeDir Directories="$(IntermediateOutputPath)$(RandomName)\"/>
    <PropertyGroup>
      <__fileToTransform>$(IntermediateOutputPath)$(RandomName)\source.$(RandomName).config</__fileToTransform>
      <__transformedFile>$(IntermediateOutputPath)$(RandomName)\transformed.$(RandomName).config</__transformedFile>
    </PropertyGroup>
    <!-- Copy original file for transformation to avoid locking issues -->
    <Copy
            SourceFiles="$(CustomConfigFileToTransfom)"
            DestinationFiles="$(__fileToTransform)"
            OverwriteReadOnlyFiles="True" />
    <TransformXml
            Source="$(__fileToTransform)"
            Destination="$(__transformedFile)"
            Transform="$(CustomConfigTransformFile)"
            StackTrace="True" />
    <!-- now, result of transformation shall be copied to original place -->
    <Delete Files="$(CustomConfigFileToTransfom)" ContinueOnError="True" />
    <Copy
            SourceFiles="$(__transformedFile)"
            DestinationFiles="$(CustomConfigFileToTransfom)"
            OverwriteReadOnlyFiles="True" />
    <!-- And intermediate results shall be deleted -->
    <Delete Files="$(__transformedFile)" ContinueOnError="True" />
    <Delete Files="$(__fileToTransform)" ContinueOnError="True" />

    <Message Text="Config after TransformCustomConfigFile" />
    <MSBuild Projects="$(MSBuildProjectFile)" Properties="_FileToRead=$(CustomConfigFileToTransfom)" Targets="_ListFileContent" ContinueOnError="True" />
    <Message Text="End of config after TransformCustomConfigFile" />
  </Target>

Final task is to use the system.website.hostname Teamcity variable and value of the umbracoPath key from web.config, to build the URL to access Umbraco with the help of an additional MsBuild task. I have to build such a cumbersome solution, because there are differences between how attribute values shall look:

baseUrl shall not contain schema, but port (e.g. it shall be something like mysite.com:80/Umbraco)
umbracoApplicationUrl shall be just fully qualified domain name (FQDN) - (http://mysite.com/umbraco)

It is done by using 2 built-in MsBuild XmlPoke tasks:

<XmlPoke Namespaces="&lt;Namespace Prefix='msb' Uri='http://schemas.microsoft.com/developer/msbuild/2003'/&gt;"
XmlInputPath="$(UmbracoSettingsConfigFullPath)"
Query="//settings/scheduledTasks/@baseUrl"
Value="$(BaseUrl)"
Condition="$(__SetScheduledTasksBaseUrl)" />
<XmlPoke Namespaces="&lt;Namespace Prefix='msb' Uri='http://schemas.microsoft.com/developer/msbuild/2003'/&gt;"
XmlInputPath="$(UmbracoSettingsConfigFullPath)"
Query="//settings/web.routing/@umbracoApplicationUrl"
Value="$(UmbracoApplicationUrl)"
Condition="$(__SetUmbracoApplicationUrl)" />

Self-hosted agents at Azure DevOps: a little cost-saving trick

Anton Kuryan — Wed, 26 Dec 2018 13:16:01 +0000

Azure DevOps does a great job when providing hosted agent services. They come loaded with all required software, they care about updates and everything else, but they have some major drawbacks:

No static external IP-address (so it's not possible to have an additional layer of security))
You get a new VM each time, so you need to clone your repository fresh, install a fresh set of NPM packages, install all those base Docker images (not a big deal for alpine-based images, but when it comes to Microsoft one, it is really a hitter)
For closed source projects there is a hard limit of 1800 minutes per month per 1 hosted job (I do not like limits even if I never hit them 😀 ) And so on – you name it

To overcome this, one can deploy self-hosted agents, but then you'd have to deal with updates, installation of tooling and extra cost. How to deal with updates and actual management is covered in this blog post, but it still leaves the question of cost partially open. I have spent some time and did several improvements to the work of Wouter de Kort in a fork of his repository, check out at my repository. Here I did some scripting optimization and improvement, but the main thing, which I wish to cover in this blog post, is the cost-optimization tool I built for our fleet of self-managed agents.

Problem

We have had only 2 hosted VS2015/17 jobs for performing builds and releases for 10 projects, each of which was requiring anywhere from 5 to 20 minutes to build and somewhere between 15 and 30 minutes to release. That was quite taxing, especially when queues built-up.

Idea

In our particular situation we have 7 parallel jobs for self-hosted agents, which comes for "free", through subscription to Visual Studio, so I began searching how to leverage those to improve our build and release speed. Initial setup was strictly following the Wouter de Kort blog series, with automated switching of VMs in a Virtual Machines Scale set nightly and on weekends to save costs. But, as soon as I began receiving requests to start some VMs for the weekend or start them later at evening to fulfill some off-hour tasks, or earlier in the morning to do urgent deployments, I started seeking for a way to automate these tasks. Which lead to having the idea of a continuous web job, which will continuously monitor the queue in the target pool and start VMs when they are needed (and stop them when they are not needed).

Realization

I came to Azure DevOps with a strong TeamCity background, so I was hoping to find something similar to a build queue in TeamCity, but, alas, they have another approach: all tasks are assigned to a pool, where they are picked up by the first agent to be online and free in a FIFO manner (first in, first out). There is no queue at all, all tasks just have a property "Result". If it is null, then this task has not yet been executed. If all tasks assigned to the pool have a non-null property "Result", then there is nothing to do. So, if there are some tasks in a pool with a null property "Result", then the code will check how much online agents are present in the pool. If the agent count is more than or equal to the tasks count, again, there is nothing to do. If the agent count is less than the tasks count, we need to start more virtual machines in the virtual machines scale set for our agents. If there are more agents online present in our pool than the number of assigned tasks, we need to stop extra agents in the virtual machines in scale set. Also, there is an option to define business hours and days of the week when there is a minimum required amount of agents online to speed up development (so, teams do not have to wait for an agent to become active and consume a task, but the task will be started immediately). The check to provision more VMs is done once every 2 minutes, to minimize waiting time and not abuse the API too much. The check to deprovision VMs is done once in 15 minutes, this allows more runtime for agents. Normally, I observe that almost immediately after a successful build a developer will wish to deploy to the Test environment.

In my humble opinion, this solution is better than statically switching off/on virtual machines on some schedule, because it allows to fulfill any task (compile, test, release, whatever is executed on your agents) at any time in a Just-In-Time manner. Though, naturally, if all agents were switched off, it will take some time for them to become online, but due to the business hour / day option, this will only happen in off-hours.

All settings and deployment instructions for the Autoscaler application are described here. I would not duplicate them in this blog post, as overtime they could change, and the readme document will be kept up-to-date.

The code of the Autoscaler app can be seen at this location.

There is also an ARM template for the baseline configuration of a web app which is suitable if you have only one pool in Azure DevOps to monitor, as it defines settings at an App Settings level of the web app itself. If there is more than one pool, only shared settings shall be defined in the App Settings of web app, while specific settings should be added to the App Settings of an individual web job. You can host as much web jobs as you need, but mind the web app limits.

Be aware, that the ARM template by default will deploy to a D1 web app (Shared web app, which allows limited amount of CPU time and only 1 Gb of RAM without Always On). The "Always On" feature ensures that the hosting process is always awake and is not shut down after 20 minutes of inactivity. So, if a web job will be deployed without additional precautions, it would not work, as the web app runtime will shut down the Kudu process after 20 minutes of inactivity. There is a nice trick to keep it up and running: you need to ping the Kudu homepage of your web app at least once every 20 minutes. I am using https://www.happyapps.io/ to visit the Kudu homepage of my web app once per 5 minutes on the address https://webappName.scm.azurewebsites.net/

Deployment hints

By default, the Azure Web app runtime does not execute the continuous web job from the path where it is deployed to, but I still wish to be sure that it is not running when I am deploying it, so I am using the following PowerShell scripts to stop/start the web job

To stop webjob:

Invoke-AzureRmResourceAction -ResourceGroupName resourceGroupName -ResourceType Microsoft.Web/sites/continuouswebjobs -ResourceName webAppName/webJobName -Action Stop -Force -ApiVersion 2018-02-01

To start webjob:

Invoke-AzureRmResourceAction -ResourceGroupName resourceGroupName -ResourceType Microsoft.Web/sites/continuouswebjobs -ResourceName webAppName/webJobName -Action Start -Force -ApiVersion 2018-02-01

The same scripts are used when rebuilding the Virtual Machines Scale set to ensure that the web job will not attempt to stop the VMs before they have been registered at the pool.

This blog post have been created by me and edited by my friend Rob Habraken.

Sitecore JSS: a blue green Node.js server configuration

Anton Kuryan — Fri, 23 Nov 2018 14:54:42 +0000

The target of this post is to share how to configure Sitecore JSS headless with a Node.js proxy on CentOS and to demonstrate a staging approach for the hosting of the CD role. Since the front-end is rendered on a stand-alone Node proxy, we can easily configure as much backend CD servers as needed, which allows us to deploy CD code on a staging server, and then switch the target server on the Node instance.

Node.js JSS server configuration on CentOS – instructions and scripts

This part will show you an example of how you can configure your Node.js SSR server from scratch. For this I assume that you have basic knowledge of working in Linux.

cd
wget linkGoesHere

Go to your local users home and download the required Node.js release (replace linkGoesHere in the following script with the link from Node.Js download page):

mkdir node
tar xvf node-v*.tar.xz --strip-components=1 -C ./node
cd
rm -rf node-v*

Create a directory, unpack the downloaded archive and remove it afterwards:

mkdir node
tar xvf node-v*.tar.xz --strip-components=1 -C ./node
cd
rm -rf node-v*

Instruct npm where the symlinks shall be put:

cd
mkdir node/etc
echo 'prefix=/usr/local' > node/etc/npmrc

Move Node.js to /opt and create the required symlinks to execute it:

sudo mv node /opt/
sudo chown -R root: /opt/node
sudo ln -s /opt/node/bin/node /usr/local/bin/node
sudo ln -s /opt/node/bin/npm /usr/local/bin/npm

When a command is executed with sudo, /usr/local/bin is excluded from the path. So, we need to add this path to the allowed paths (to have the possibility to call node and npm with sudo):

sudo visudo

Find the line that specifies Defaults secure_path and add :/usr/local/bin to the end of it.
should look like
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

We've just finished the installation of node.js . To verify this, execute node -v, which will show you the installed Node.js version.

To manage the SSR proxy and keep it up and running, we need a Node.js process manager. *nix world has a lot of them, so it can be difficult to pick one. I chose to use pm2, because it is simple, easy to use and has an option to either watch or ignore file system changes.

So let's install and daemonize pm2:

sudo npm install pm2@latest -g
#I want daemonize node app via current user, not root
#follow instructions of command
pm2 startup systemd

After that, pm2 will start its service under the current user.

Since Node.js apps is not big in security and caching (without special precautions), and setting up SSL on them is also quite a challenge, I will put Nginx in front of it to listen for requests on ports 80 and 443, and send them to my SSR proxy.

To install Nginx, execute the following script:

sudo yum install epel-release
sudo yum install nginx
sudo /usr/sbin/setsebool httpd_can_network_connect 1

And ensure that Nginx does not expose its version to improve system security (we should disclose a minimum amount of data about our systems):

sudo vi /etc/nginx/nginx.conf
#Then, inside the http configuration part add the line server_tokens off; like this:
http {
        server_tokens off;

Then, configure SSL:

sudo mkdir /etc/nginx/ssl/
sudo openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096

Put your certificate and key into "/etc/nginx/ssl" and note the paths.
Subsequently, restore the security configuration for the SSL certificates:

restorecon -v -R /etc/nginx/ssl

And configure your application. Nginx will redirect all incoming requests on port 80 to port 443; on port 443, after a successful TLS handshake, all requests will be deciphered and sent to the Node.js app, hosted in pm2 and listening on port 3000:

sudo touch /etc/nginx/conf.d/app.conf
sudo vi /etc/nginx/conf.d/app.conf

server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;
   return 301 https://$host$request_uri;
}
server {
  listen 443 ssl http2 default_server;
  listen [::]:443 ssl http2 default_server;
  server_name _;
  root /usr/share/nginx/html;
  ssl_protocols TLSv1.2;
  ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
  ssl_prefer_server_ciphers on;
  ssl_dhparam /etc/nginx/ssl/dhparam.pem;
  ssl_certificate /etc/nginx/ssl/YOURCERTIFICATE;
  ssl_certificate_key /etc/nginx/ssl/YOURKEY;
  location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-Port 443; 
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Now check the configuration, restart Nginx and enable it:

sudo nginx -t
#if there is errors here – fix them
sudo systemctl restart nginx
sudo systemctl enable nginx

Clone the Sitecore SSR proxy example and prepare for hosting it in PM2:

cd
#SSR proxy configuration
git clone https://github.com/Sitecore/jss.git
sudo mkdir /opt/node-app/
sudo chown YOURUSER: /opt/node-app/
cp jss/samples/node-headless-ssr-proxy/** /opt/node-app/
cd /opt/node-app/ && npm install

At this point, the application is still not functioning, but all further configuration will be done on VSTS.

VSTS release configuration to deploy to staging server and then switch it

My release pipeline consists of several stages, but this blog post will not cover it all, as it would take a lot of time. Instead, I will concentrate on explaining how to do "green/blue" deployments for JSS.

The main tools used during a release are an SSH connection to deliver changes to the CentOS host and MsDeploy to deliver changes to IIS.

Our configuration is pretty simple: we are running on Sitecore 9.0.2 with JSS; our infrastructure consists of 1 CentOS webserver for the Node.js SSR proxy, 3 Windows servers (1 for CM, 1 for xConnect and 1 for the CD role). On the CD server I've configured 2 IIS websites, which names are the same as internal hostnames (cd1.host.local and cd2.host.local).

CD and Node.js delivery stage

The first step of this stage is to check the Node.js proxy configuration, to understand on which server we should deliver our code now (so we are deploying to a website which is not under load ATM). It is done by connecting to our Node.js webserver and grabbing the hostname from the config (and since the hostname is the same as the IIS site name, that's all I need from there). Then this value is stored in a variable to be reused by subsequent steps:

currentHost=`grep -A0 'apiHost' $(nodeApp.Path)/config.js | grep -oP "http(s)?:\/\/(.*)'"|sed 's/https\?:\/\///' | sed "s/\(.*\).\{1\}/\1/"`

echo "##vso[task.setvariable variable=deploy.currSite;]$currentHost"
deploymentHost=`[[ "$currentHost" == "cd1.host.local" ]] && echo "cd2.host.local" || echo "cd1.host.local"`

echo "##vso[task.setvariable variable=deploy.iisSiteName;]$deploymentHost"

The variable "nodeApp.Path" contains the value "/opt/node-app", which is where we've deployed the node-headless-ssr-proxy sample from https://github.com/Sitecore/jss.git to, and the "deploy.iisSiteName" variable will now be holdingthe name of the target website (the one which is not under load). And then, even at first run, this script will not fail.

"$(msdeploy.Path)" -allowUntrusted="True" -verb:sync -source:recycleApp -dest:recycleApp="$(deploy.iisSiteName)",computerName="https://$(deploy.targetComputer):8172/msdeploy.axd?site=$(deploy.iisSiteName)",recycleMode="StopAppPool",username="$(user)",password="$(password)",authType="Basic"
"$(msdeploy.Path)" -allowUntrusted="True" -enableRule:DoNotDeleteRule -verb:sync -source:package="%sourcePath%" -dest:contentPath="$(deploy.iisSiteName)",computerName="https://$(deploy.targetComputer):8172/msdeploy.axd?site=$(deploy.iisSiteName)",username="$(user)",password="$(password)",authType="Basic"
"$(msdeploy.Path)" -allowUntrusted="True" -verb:sync -source:recycleApp -dest:recycleApp="$(deploy.iisSiteName)",computerName="https://$(deploy.targetComputer):8172/msdeploy.axd?site=$(deploy.iisSiteName)",recycleMode="StartAppPool",username="$(user)",password="$(password)",authType="Basic"

The next steps will stop the app pool, deploy a prepared application package and start it (to allow the CD server to preheat itself with an application init) via MsDeploy with the following commands:

The next step is to update the Node.js app on the CentOS host. /dist/app content is prepared during web app compilation, so here we only need to update config.js with the new hostname. I am using https://github.com/qetza/vsts-replacetokens-task#readme to replace the token "deploy.iisSiteName" in config.js.

if [ -d $(nodeApp.dist.path) ]; then rm -Rf $(nodeApp.dist.path); fi

As soon as the files are ready to upload, I execute the following SSH script to cleanup the target path:

"nodeApp.dist.path" can contain a unique value, generated from the static app name and the release number for example.

Then I will deploy the files to the CentOS host via the Copy files Over SSH task and restart the pm2 application:

cd $(nodeApp.Path)
#see https://github.com/Unitech/pm2/issues/325#issuecomment-281580956 for explanation
pm2 delete -s $(nodeApp.Name) || :
pm2 start index.js --name $(nodeApp.Name)

At this point, our Node.js SSR proxy is pointing towards the newly updated IIS website. It then could be wise to stop the app pool of the 'old' CD IIS website with the following command:

"$(msdeploy.Path)" -allowUntrusted="True" -verb:sync -source:recycleApp -dest:recycleApp="$(deploy.currSite)",computerName="https://$(deploy.targetComputer):8172/msdeploy.axd?site=$(deploy.currSite)",recycleMode="StopAppPool",username="$(user)",password="$(password)",authType="Basic"

And that's it. You have now created a simple and effective blue green setup for your headless Sitecore JSS website using a Node.js proxy and multiple CD role servers to do the swapping.

This blog post have been created by me and edited by my friend Rob Habraken

Sitecore Deployer

Anton Kuryan — Tue, 15 May 2018 08:14:37 +0000

While working with Sitecore deployments based on ARM and PowerShell scripts, developed by Rob Habraken (Rob has covered them in a series of blog posts on his own blog) and optimized by me, I noticed that we are having a problem: these scripts are used on a per-project basis and once implemented, they start aging. Since manually updating scripts for each project is a dull and cumbersome task, it usually is never executed. So I decided to build a VSTS release task, which could be reused in each of our projects and makes it easier to always have the latest version of the ARM and PowerShell scripts for a certain Sitecore version.

How it works

You can check the PowerShell script at my repository.

This extension uses a VSTS connection endpoint to authorize against Azure and will take the following input:

ARM template path
The ARM template published by the artifact source. Usually, it is a build.
ARM parameters path
The ARM parameters file for the template, published by the artifact source. They will be parsed at runtime and, in case of the ‘Deploy’ deployment type, combined with the output of the ‘Infrastructure’ deployment type and fed into template.
Resource group name
The name of the targeted resource group, where the resources will reside. If the deployment type is ‘Infrastructure’ and the resource group doesn’t exist, the script will attempt to create it, but for the deployment type ‘Deploy’ it will throw an error.
Location
A dropdown with the available locations for your resources.
Deployment type
There are several differences between deployment types: the ‘Infrastructure’ deployment type will attempt to create the resource group if it doesn’t exist and provision the infrastructure, while the ‘Deploy’ deployment type will get the output of the ‘Infrastructure’ deployment type and combine it with the parameters in the template parameters file to deploy the application.
Generate SAS?
If this is set to Yes, then the script will try to generate a short-lived SAS for the SCWDP packages, passed in the parameters. The storage account must belong to the same subscription where we are deploying to. If it is not possible to generate a SAS, it will silently continue. If the license.xml file has to be retrieved from a URL and if it is stored on a storage account which belongs to the current subscription, the script will try to generate a SAS for it as well.
License location
A dropdown which allows you to select how the license will be passed to the Sitecore installation. It could be a URL or be specified inline (e.g. copy-paste from the actual file into the field, or, which is more secure, store the value in a protected build variable and add a reference to it here). But it could also be a reference to the file system (published as an artifact of a build), and finally, you can say that your license.xml file is pasted as a parameter into your template parameter file.

Section “Security”

It is advised to deny access to the Processing role for everyone and allow access to the Reporting role for Azure only. Also, I checked with Sitecore and by default you do not need to deploy any custom code to these roles, which can speed up your deployment greatly. So, those checkboxes are needed only in case you are not deploying your own custom code on the Processing and Reporting roles and you are not running in ASE (as ASE allows to deploy additional firewalls in front of your web application).

Limit access to PRC role by IP
This one is a “catchy” setting. If you will check Sitecore templates and SCWDP packages from 8.2 till 9.0.1, you’ll notice that in all fully-fledged deployments you are allowed to set a client IP and client IP mask to limit access to the CM and PRC instances. MsDeploy (SCWDP) packages will have a field for it. By default, it is 0.0.0.0/0.0.0.0. Problem is, this parameter allows you to specify one, and only one IP, which is weird, because you’ll need at least two: 127.0.0.1 to allow applicationInitialization, fired by the Web app itself to complete initialization correctly, and your own IP to access the CM. IP and mask, specified in the relevant parameters, will be passed to the web.config and will be written in the section . And here comes another catch, because the section is defined as follows: “”, and this “clear” node removes any rules defined on the web app in the Networking section of the Azure Web app. So, the only possible solution to protect the PRC web app with only default SCWDP packages deployed (like PRC role + modules) is to inject the parameters IP 127.0.0.1 and mask 255.255.255.255. This will effectively deploy the same limitation on the CM web app, but, at the end, normally you are deploying your own set of config files on the CM instance anyway.
Limit access to REP role by IP
Following Sitecore best practices, it is recommended to limit access to reporting role by IP, allowing only Azure IPs. However, I see it as this: we need to allow access to reporting role from other roles of deployed solution (CM, CD, PRC) and, possibly, from our own IP(s). Enabling this checkbox will:
a) show additional control for entering IP/Mask in comma-separated string
b) on execution of script – all outgoing IP addresses of other roles will be added to allow list (along with 127.0.0.1 and anything entered in additional control)
Since the REP SCWDP package does not have ipSecurity embedded, this works flawlessly: IP based limitation will be deployed on web app level (you can see it in the Networking section of the Azure web app).

That’s all I wanted to describe about this new extension. Feel free to contact me for additional questions, push PRs and discuss it on the repository.

Saving money with Azure Costs Saver VSTS extension

Anton Kuryan — Thu, 03 May 2018 08:11:52 +0000

Since I started working with Azure Web apps I started thinking: how could we optimize budget spending for PaaS? For VMs it is clear, you stop them. VM v1 requires an additional PowerShell command to deprovision them to stop spending money on them, while VM v2 are deprovisioned automatically on stop. But for or Azure SQL databases and Azure Web apps it was not that straightforward.

So, I took some time to investigate it. The main issue was that I didn’t want to do cumbersome setups like passing resource group names, names of web apps, VMs, SQL databases and their sizes, while keeping track of all changes. Since we are using resource groups to group resources on environment basis, I found out that I could filter all resources from a resource group and find those that could be downsized or deprovisioned (web apps and SQL databases could be downsized, VMs deprovisioned) without hurting the overall application. For example, deleting a Redis cache will require updating the application setup on its recreation.
I feel that using a plain PowerShell script is difficult to share and is just too simple way of solving issues. So I decided to build a VSTS extension for it (TeamCity meta-runner will follow some day too). Check out the result of my Azure Costs Saver extension.

How it works

You can find the core of the extension at my GitHub account.

Downscales App service plans to S0 size with 1 runner. Currently, it doesn’t have a built-in check if a web app has a staging slot, and you cannot downscale to B0 if you have a staging slot. Also, the B-tier has a size limitation of 10 Gb of disk space, while the S-tier allows up to 50 Gb. Before downscaling, the current parameters are saved on the resource as tags.
Downscales SQL databases to S0 size, saving the current parameters on the SQL server as tags.
Downscales Elastic SQL Pools
Deprovisions VMs (both single and VMSS).

When upscaling:

For Web apps: the tags are read from the App service plan and they are upscaled back to the original sizes and number of workers written in tags.
For SQL databases: the tags are read from the SQL server and the SQL databases are upscaled back to the required sizes.
VMs are started again (those at VMSS as well)

How it could be set up

Since running something manually is not the thing I wish to do, let alone several times in a row, I decided to set this up as a VSTS Release; though, the extension could be used in Build as well.

My release is scaling resources up or down in a resource group, corresponding to 2 environments: T (test) and A (acceptance). The release is scheduled to be automatically created at 13:00 each day from Monday till Friday:

The release consists of 2 environments: one scales down and is called Minifying, which is executed once after release creation at 22:00 from Monday till Friday; the other is called Reverting and is executed once after release creation at 06:00 from Monday till Friday. This setup ensures that at night and during weekends our resources are at minimal size, while during daytime they are executing at full power.

Each VSTS environment consists of 1 phase with 2 tasks to scale up or down resources from the T and A resource groups; each task is an Azure Cost Saver task.

Fields and theirs meaning in task:

Display name: the name of a task for your convenience
Azure Connection Type: this set up to be Azure Resource Manager now.
Azure RM Subscription: a dropdown with subscriptions either accessible to you or configured via VSTS endpoints
Downscale resources?: a dropdown with a “Yes” and “No” option. If you select “Yes”, the resources will be downscaled and the current sizes (before downscaling) will be written to tags; if you select “No”, the resources will be upscaled to the sizes stored in tags.
Resource group name: the name of the resource group where the applicable resources are residing.

Conclusion

Using my Azure Costs Saver VSTS extension, you can easily save some costs within your Azure subscriptions for non-production environments, only using the full performance when required – fully automated.

Automating VSTS extensions publishing via TeamCity

Anton Kuryan — Wed, 18 Apr 2018 15:23:16 +0000

First of all, word of warning:
Be extra careful, when creating publisher in https://marketplace.visualstudio.com/ if you are member of several AAD - it is not as straightforward as one could expect. I ended up with one publisher, created from one of my AAD tenants (despite of being authorized at MSFT account - there is a dropdown to select tenant). This is a problem - I have not had VSTS account with these tenant, so I was not able to automate things (I created token, but token was from MSFT user, not AAD tenant user). I ended up requesting support for help - they added my MSFT login to my publisher.

Now, step by step guide on how to setup Teamcity build.

1) Install Teamcity.Node plugin

2) Extension version are patched on Teamcity level (so, developer does not need to keep in mind necessity to bump up those numbers, though, he have to inform DevOps or update Major version in case there is breaking changes)
I am using "VCS labeling" feature of Teamcity to put tag of successful builds and "File content replacer" feature to replace versions in vss-extension.json and task.json. This way, Teamcity will always bump up version in both files before creating extension vsix package.

I was experimenting with several regexps (so, this is not single point of truth, but useful start for your own implementations)

3) Build steps:

Node.js NPM from Teamcity.Node plugin It installs tfx-cli and updates it with following commands: "install -g tfx-cli up -g tfx-cli"
Command line It invokes tfx to publish my extension "tfx.cmd extension publish -t %access.token% --manifest-globs vss-extension.json --no-color --no-prompt"

Some additional findings

When you need to generate a PAT (personal access token) described https://docs.microsoft.com/en-us/vsts/extend/publish/command-line?view=vsts - do not forget to select "All accounts" in dropdown (as indicated here - https://social.msdn.microsoft.com/Forums/vstudio/en-US/21073d22-f16e-40d2-9568-3d379e64ff00/why-my-token-cannot-be-used-and-it-always-return-401-error?forum=vsx ).
Fix Windows search logic for NPM extension - I was not able to start it via %AppData%\npm\tfx from Teamcity, so, I created a symbolic link for %AppData%\npm for my agents running under System account (mklink D:\data\agentNpm C:\Windows\System32\config\systemprofile\AppData\Roaming\npm /d) and executed it in command line runner as D:\data\agentNpm\tfx.cmd extension publish -t %access.token% --manifest-globs vss-extension.json --no-color --no-prompt

Trusted self-signed certificate on local Windows machine

Anton Kuryan — Tue, 13 Mar 2018 13:37:03 +0000

This is cross-post from my blog.

I created powershell script to automate generation of locally trusted self-signed certificate. It will create root certificate, to be used as signing certificate for future generation of other certificates.

Please, see script at my repository here. I hope it will help.

After certificates have been generated - you can share them with your developers: export as PFX certificate for your domain, and export as CER file certificate from LocalMachine/Root which is issue to "DONOTTRUST-LocalSigningCertificate". Your fellow developer shall import CER file (certificate without key) to "LocalMachine/Root", PFX - to "LocalMachine/My" and can work with self-signed locally trusted certificate happily (NOTE: you could not use this certificate in open, as it would not be trusted by default - that's only for development purposes to save time and money)