DEV Community: Jeremy Davis

Reformatting config XML so it’s easier to diff

Jeremy Davis — Mon, 30 Sep 2019 07:22:37 +0000

Every so often pretty much every developer ends up in a situation where they’re looking at a bug that manifests on one platform, but not on another. The sort of bug where you end up spending hours looking through log and config files for a subtle difference. I found myself looking into just this sort of bug recently, but on a site where (to my frustration) the config files were full of comments and whitespace differences across platforms that made diffing really hard**. Spotting that subtle bug-causing difference is pretty much impossible when your diff is full of noise… So how can we fix that?

With some Powershell of course!

I realised life would be much simpler for me if I could standardise the formatting and drop all the comments from the config. That way any differences that the diff tool showed up would be actual configuration differences, and not any of the annoying noise. So I hacked up some code to iterate a tree of .config files and reformat them for me…

Via the System.Xml.XmlDocument object we can load a config file while ignoring comments, and then write it back to disk with standardised indenting with a function like this:

function tidyConfigFile
{
    param(
        [string]$file
    )

    Write-Host "Modifying $file"

    $settings = New-Object System.Xml.XmlReaderSettings
    $settings.IgnoreComments = $true

    $reader = [System.Xml.XmlReader]::Create($file, $settings)

    $xml = New-Object System.Xml.XmlDocument
    $xml.Load($reader)

    $reader.Dispose()

    $xml.Save($file)
}

The only thing of note there is that you need to use an XmlReader with special settings to get the XmlDocument to strip out comments when it’s loading.

So that will take some messy xml like

<?xml version="1.0" ?>
<!-- hello -->
<alpha>



<test x="y"/>

</alpha>

and neaten it up to:

<?xml version="1.0"?>
<alpha>
  <test x="y" />
</alpha>

So little recursive function to step through a tree of folders updating any config files will sort out the rest of the task:

function processFolder
{
    param(
        [string]$directory
    )

    Write-Host "Processing $directory"

    $files = Get-ChildItem $directory -Filter *.config | Select -ExpandProperty FullName

    foreach($file in $files)
    {
        tidyConfigFile $file
    }

    $children = Get-ChildItem $directory -Directory | Select -ExpandProperty FullName

    foreach($child in $children)
    {
        processFolder $child
    }
}

So with that in place, I could just put copies of my two sets of config in one folder, and let the script go…

processFolder ".\CopiedConfig"

To get myself a set of files that were much easier to diff.

Simple…

** Yes, I know full well the fact there are differences like this is a massive code smell. But sometimes when you're the support developer on projects with little budget you just have to hold your nose and make it work, rather than spending days reworking the entre deployment process...

Caching when you have duplicate container components

Jeremy Davis — Mon, 16 Sep 2019 07:10:33 +0000

In theory, the magic of Dynamic Placeholders lets us have a container component placed onto your page more than once. That didn’t work in the old world of “static” placeholders, because the rendering engine didn’t like two placeholders with the same name. But despite it’s benefits, the dynamic implementation has an annoying edge case – you may not be able to enable caching for your container component. I had a client bump into this issue recently, so I spent some time considering approaches that might help them address this issue.

The issue

Imagine you have a container component. A very simple one will do fine – all it really needs to contain is a Dynamic Placeholder:

@using Sitecore.Mvc
<div style="border:2px solid red; margin-top:10px; padding:10px;">
    <div>Dynamic Container</div>
    @Html.Sitecore().DynamicPlaceholder("container")
</div>

If you build up a page using two instances of that component, with some child components inside each of the placeholders it will look fine in Experience Editor:

But if the container components have caching enabled, it won’t look right on the public site:

That’s because the code renders the first container and caches the result. Then it starts to render the second one, but gets a cache hit, and so returns the cached data for the first one. In the cache you see:

There’s no cache variant that works for “this component varies by its children” – so only the first instance of the container ever gets rendered and cached.

My interest in this came about because I got handed a bug report on a client site that said “our page is wrong!” and it turned out to be because of this issue. I think there are there main ways that the issue could be resolved to make the client happy:

Solution one: Just disable caching

If you turn the cache settings off for the container component, all will look right.

That’s probably fine if your container has few children, or it’s not used very much. The performance difference in that scenario is likely to be small – especially if you are able to enable caching for the child components inside your container.

That’s a fix developers can roll out easily (it’s just undoing the cache settings on the rendering item) and it doesn’t affect editors at all.

Solution two: Use a fake data source

You could configure the container component with the “vary by datasource” cache setting instead of turning caching off. If the component doesn’t really need a data source, the code will just ignore what ever item the datasource setting points to, other than for caching. And that can allow the normal caching framework to work here.

No deployment is required for this, which is a bonus. But it might be confusing for editors to need to add a data source where it’s not really needed. Plus this doesn’t work if your container does need a datasource, and you have two on a page that need the same datasource but different children.

So this might not be a great answer.

Solution three: Vary the cache by rendering id

Finally, it turns out it’s not actually that hard to extend Sitecore’s caching model to allow for the idea of “cache this specific rendering instance”. You could write some code that looked at the actual children of a component and generated a cache key based on that. Maybe by hashing some data from the Rendering XML? But it struck me that it’s faster and easier (and probably much the same outcome) to just make use of the unique ID that every instance of a rendering gets by default…

To do this, you need to create a new data template to store your “cache by rendering’s unique id” flag:

And then add that to the inheritance tree of the standard caching flags item:

And that lets you set a caching flag on your container rendering:

To act on that new flag, you need to deploy a simple bit of code to extend the rendering pipeline:

public class VaryByUniqueIdCacheKey : Sitecore.Mvc.Pipelines.Response.RenderRendering.GenerateCacheKey
{
    protected override string GenerateKey(Rendering rendering, RenderRenderingArgs args)
    {
        var cacheKey = base.GenerateKey(rendering, args);

        var cacheField = (CheckboxField)rendering.RenderingItem.InnerItem.Fields["VaryByUniqueId"];

        if (cacheField.Checked)
        {
            cacheKey += "_#uniqueId:" + rendering.UniqueId.ToString();
        }

        return cacheKey;
    }
}

That looks for the extra cache variant flag on the item being rendered, and if it exists the cache key is extended with the unique ID for this rendering instance.

And you can patch that into your config via:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <mvc.renderRendering>
        <processor patch:instead="processor[@type='Sitecore.Mvc.Pipelines.Response.RenderRendering.GenerateCacheKey, Sitecore.Mvc']" type="Caching.VaryByUniqueIdCacheKey, Caching" />
      </mvc.renderRendering>
    </pipelines>
  </sitecore>
</configuration>

Once that’s done, you’ll get a separate cache entry for each instance of your container component:

And the published version of the site will look correct:

Whether that solution is “better” for you than just disabling caching probably depends on the effort involved in rendering each of your containers. If they have one or two items in them, chances are it’s not worth it. But if your container had a larger number of items, the “memory usage vs processing time” trade-off might be worth it.

It’s an option for you…

An edge case of remote events with Publishing Service

Jeremy Davis — Mon, 02 Sep 2019 07:23:21 +0000

I’ve been working on an international deployment of Sitecore recently, and resolving some problems around how publishing raises remote events has demonstrated that there are some things about the publishing process that I didn’t entirely understand… I doubt this is a common scenario, but it still seems worth writing down what I’ve learned – So here’s another crib sheet for my future self:

The scenario:

I won’t go into the reasons for why, but this deployment has Sitecore broken up into three bits:

A shared Master database that lives in a central location
A Sitecore cluster in Europe which has all the roles other than the Master DB
A Sitecore cluster in Australia which has all the roles other than the Master DB

So authors log in to the Sitecore instance that’s geographically closest to them, and do their editing there. When they’re done they hit “publish” and the changes are sent to Publishing Targets for both the EU and AU web databases.

Since there’s a WAN link between the central Master DB and the local Web DBs, this seemed like a use-case for Publishing Service, as it’s much faster processing should be much better at dealing with the network latency involved in this scenario.

And mostly this works ok…

But it was noticable that when an author published in Europe, it updated fine, but caches and indexes did not update in Australia. And that made sense, as checking under the surface the Events tables in the Australian databases did not get any “a publish has happened” messages written to them…

So clearly I got something wrong – and after failing to fix it myself I spent some time discussing the scenario with Sitecore Support.

My bad assumptions…

Initially I looked at the role diagram for the site, and thought that Publishing Service instances need to know about both the EU and AU databases, but they are the only bits of the system that need to.

So Publishing Service can be configured to see the “web” Publishing Target databases in EU and AU, plus the central Master. But the CM and CD instances only care about their local web database and Master. So they don’t need to know about the remote web databases at all.

That seems logical – and the publish operations largely work. But the “events don’t happen remotely” thing suggested it wasn’t right.

What I learned from support…

So it turns out that the “a publish has happened” events are not actually generated by Publishing Service. It does the work of moving items between databases, but it doesn’t do the work of triggering all the events. That work still happens inside the CM instance.

That means that this scenario requires you have to have all your remote databases configured as Publishing Targets for both the CM role and for Publishing Service.

So fix my issue, I had to change the configuration so that:

All of the web databases appear in the CM server connection string files with unique names. e.g. naming them “web_europe” and “web_australia”
All of the web databases need to be configured as Publishing Targets in both of the CM instance, using these unique names. This requires both a config patch to configure the event queues etc, and making sure there are the right items under “/system/Publishing Targets” in Master that define these publishing targets, using the unique database names.
The definitions on the CM box need to have their “database” attributes updated to use the unique database names.
The configuration for the target databases in Publishing Service must use the same unqiue names, and must specify the unique database names for each target. (So Publishing Service needs these databases in its connection strings file too)

Once both Publishing Service and all of the CM boxes knew about all of the databases, the events started to appear in the remote database after a publish, and magically caches cleared and indexes updated after each publish.

You don’t appear to need to change the CD server config to match the CM changes.

Once again, Sitecore Support saves my bacon – Thank you!

Clearing up a little confusion with Let’s Encrypt’s DNS challenges

Jeremy Davis — Mon, 19 Aug 2019 08:00:48 +0000

I love Let’s Encrypt. I’ve been using SSL for all my personal projects for years. Until they came along I was using self-signed certificates and manually adding my own root certificate to all the machines I was using my sites from. And that lead to some fun… So I moved eagerly to Let’s Encrypt when the tooling supported Windows reasonably well, and set myself up with a certificate with multiple SANs authenticated via their “HTTP proofs” mechanism, and it all worked fine, despite it being a bit of a pain that I had to expose port 80 for sites I only wanted accessible via port 443.

But I realised recently that they now offer wildcard certs that would make my life simpler, and that there is now decent support for DNS-based proof-of-ownership. So recently I tried moving my server over to this model – and there was a bit of friction. Entirely PEBCAK though – so I’m writing this down for the next time I forget how DNS works

As a user of Windows servers, the Certify client is the easiest way to set up this sort of thing. So I started by downloading the latest instance of that – which worked easily enough.

From there you can request a new certificate for “*.yourdomain.com”. You also have to tell it that you’re using a DNS challenge. Depending on your DNS provider, you may be lucky enough to have support for automatically updating DNS with the challenge data. That worked for me with one domain, which was hosted via CloudFlare. But it wasn’t available for me with another domain – which is where I had a little confusion.

If automation isn’t supported for your DNS provider you have to select the “manual” update mechanism… Then, when you request the certificate, Certify tells you about the TXT record(s) you need to create. You get something like this from the UI and logs:

2019-06-22 14:24:06.261 +01:00 [INF] DNS: (Update DNS Manually) :: Please login to your DNS control panel for the domain '\*.mydomain.co.uk' and create a new TXT record named: \_acme-challenge.mydomain.co.uk with the value:a-ELrRGO72uAYDPxb8YpyzDrHkC8n71Yc67sW7HX1bX

So you log in to your DNS provider, and copy/paste these values into a TXT record…

Except no…

Don’t make the silly mistake I did intially – remember how DNS works…

You do not log in to your DNS portal and paste “_acme-challenge.mydomain.co.uk” into a new TXT record’s Name field – you create a TXT record named “_acme-challenge” under your domain “mydomain.co.uk”:

Seems very obvious in retrospect, but it this caught me out because it seems some DNS providers will recognise the mistake here, and strip the DNS suffix off the name of your new TXT record. Cloudflare, for example, are wise to this mistake. But some will not – including my other provider.

And if you do make this copy/paste mistake the Let’s Encrypt validation will not work, because it will not be able to find the “right” TXT record. And then you’ll spend half an hour carefully comparing the challenge value, wondering if you got that random string of nonsense wrong…

Must remember not to make this mistake next time

Another Marketing Automation gotcha

Jeremy Davis — Mon, 22 Jul 2019 07:30:05 +0000

Following on from my recent post about how I was able to mess up my life by getting Marketing Automation connection strings wrong, I hit another interesting issue with MA – this time around content languages…

The issue

Looking at a newly deployed instance of Sitecore, my editor account couldn’t see any Campaign Templates in the Marketing Automation dashboard:

That wasn’t right, as I could see the template items sitting in Content Editor:

Doing a bit of digging, I noticed that that dasboard page was making a web service call that I could examine via browser dev tools:

The request was passing in a querystring value: “cultureName=fr-FR” And the result of that call was a valid json response for no data:

[]

But if I changed that url to use “en” as its language, the data changed:

[
  {
    "id": "938ec74b-3ba4-4c93-9152-015655088998",
    "alias": "938ec74b-3ba4-4c93-9152-015655088998",
    "name": "Cart Abandonment",
    "children": null,
    "startDate": "0001-01-01T00:00:00Z",
    "endDate": "0001-01-01T00:00:00Z",
    "createdBy": "sitecore\\Admin",
    "createdDate": "2018-08-17T12:26:31Z",
    "cultureName": "en",
    "customValues": {},
    "description": "When a customer fails to complete an order, an automated marketing email message  is sent to the recipient. The intention is to re-engage the recipient and to encourage them to complete the order. Appropriate timings differ depending on the sales cycle of the product. In our example we show three email messages over three days. To discourage misuse of promotional codes in this campaign, brands may want to set a limit to the number of times a recipient can re-enter this campaign.",
    "isActive": false,
    "state": "Inactive",
    "lastModifiedBy": null,
    "lastModifiedDate": "2018-10-05T06:54:16Z",
    "version": 1,
    "classifications": {},
    "currentEnrollments": 0,
    "marketingThemeId": "85506484-7854-4b14-ab0c-dfc61ca65cc0",
    "maxEnrollmentsCount": null
  },

  -- snip --

  {
    "id": "ae5178d0-9484-41f3-8c9e-cb1b8f478e24",
    "alias": "Blank",
    "name": "Blank",
    "children": null,
    "startDate": "0001-01-01T00:00:00Z",
    "endDate": "0001-01-01T00:00:00Z",
    "createdBy": "sitecore\\Admin",
    "createdDate": "2018-04-03T04:07:59Z",
    "cultureName": "en",
    "customValues": {},
    "description": "",
    "isActive": false,
    "state": "Inactive",
    "lastModifiedBy": null,
    "lastModifiedDate": "2018-04-03T04:09:06Z",
    "version": 1,
    "classifications": {},
    "currentEnrollments": 0,
    "marketingThemeId": "00000000-0000-0000-0000-000000000000",
    "maxEnrollmentsCount": null
  }
]

So the issue is something to do with languages…

Working around the issue

Thinking about this, I tried looking at the user’s settings:

And I realised that these users who were having problems were set to have a specific default content language to match the client’s project requiremenents. If I change the user back to having the “default” or “en” then the Marketing Automation dashboard starts to work correctly again – which would fit with the behaviour seen by changing the url above.

A colleage suggested translating the Marketing Automation template items into the client’s required language might make a good workaround, so I tried that approach. Any items from the templates folder above that you translate start to appear in the templates view. So you can use the “create from template” option to try and make a new Marketing Automation flow. But unfortunately when the editor loads up you still get an error:

In the logs, you get:

 728 20:37:07 ERROR [Sitecore Services]: HTTP GET
URL http://sc911.sc/sitecore/api/ma/automationcampaigntemplates/422e238c-65aa-4ec9-8f98-d563cdee6c72?cultureName=fr-FR&sc_site=shell

Exception System.Resources.MissingManifestResourceException: Could not find any resources appropriate for the specified culture or the neutral culture.  Make sure "Sitecore.Xdb.MarketingAutomation.Locators.Xmgmt.Resources.resources" was correctly embedded or linked into assembly "Sitecore.Xdb.MarketingAutomation.Locators.Xmgmt" at compile time, or that all the satellite assemblies required are loadable and fully signed.
   at System.Resources.ManifestBasedResourceGroveler.HandleResourceStreamMissing(String fileName)
   at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(CultureInfo culture, Dictionary`2 localResourceSets, Boolean tryParents, Boolean createIfNotExists, StackCrawlMark& stackMark)
   at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo requestedCulture, Boolean createIfNotExists, Boolean tryParents, StackCrawlMark& stackMark)
   at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo culture, Boolean createIfNotExists, Boolean tryParents)
   at System.Resources.ResourceManager.GetString(String name, CultureInfo culture)
   at Sitecore.Xdb.MarketingAutomation.Locators.Xmgmt.AutomationPlans.ItemActivityDescriptorLocator.ToActivityDescriptor(ActivityTypeItem activityTypeItem, CultureInfo culture)
   at Sitecore.Xdb.MarketingAutomation.Locators.Xmgmt.AutomationPlans.ItemActivityDescriptorLocator.GetDescriptor(Guid id, CultureInfo culture)
   at Sitecore.Marketing.Automation.Extensions.AutomationActivityDefinitionViewModelExtensions.ToViewModel(IAutomationActivityDefinition automationActivityDefinition, IActivityDescriptorLocator activityDescriptorLocator, CultureInfo culture, IActivityConverterFactory`1 activityConverterFactory, BaseLog logger)
   at Sitecore.Marketing.Automation.Extensions.AutomationPlanDefinitionViewModelExtensions.<>c__DisplayClass0_0.<ToViewModel>b__2(IAutomationActivityDefinition x)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)
   at Sitecore.Marketing.Automation.Extensions.AutomationPlanDefinitionViewModelExtensions.ToViewModel(IAutomationPlanDefinition automationPlanDefinition, IActivityDescriptorLocator activityDescriptorLocator, CultureInfo culture, IActivityConverterFactory`1 activityConverterFactory, BaseLog logger)
   at Sitecore.Marketing.Automation.Data.AutomationCampaignTemplateRepository.GetById(Guid id, String cultureName)
   at Sitecore.Marketing.Automation.Client.Controllers.AutomationCampaignTemplatesController.<GetById>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.TaskHelpersExtensions.<CastToObject>d__1`1.MoveNext()

That reads like there’s a binary which does not have resource file data for the French language – and hence it’s not going to work.

However if I do switch the user back to having their default language be “en” then this does work… So it looks like users who want to modify automation flows need to make sure they have a default content language set “en” while they’re working…

Shooting myself in the foot with Marketing Automation

Jeremy Davis — Mon, 08 Jul 2019 08:05:30 +0000

I had another of my fun chats with Sitecore Support recently, for an issue that seemed to get no useful results in Google when I looked. So, as is my way, I’m filling that search-engine void today. This turned out to be entirely my fault – but it seems like the sort of mistake that others might encounter too… So if you’ve deployed a distributed instance of Sitecore and found Marketing Automation was behaving oddly, read on…

The issue

I’ve been working on a large deployment of Sitecore v9.1 Update 1, using the XP1 SIF scripts. Mostly this has been pretty pain-free. I was able to edit the assorted json files to fill in the specific bits of configuration I needed for my deployment, and then run the right scripts in the right places to make it all work.

But after the initial deployment I noticed that one aspect of the site was not working correctly. When you tried to browse to the Marketing Automation UI, you’d get a pause and then a cryptic error:

Digging into the Sitecore logs on my CM server, I found this error, sitting on top of a very long stack trace. I’ve trimmed off the end, as it just repeats much the same thing for a long time, but the interesting bit was:

13976 14:53:30 ERROR [Sitecore Services]: HTTP GET
URL https://author.redacted.com/sitecore/api/ma/plans/?cultureName=en-GB&sc_site=shell

Exception System.InvalidOperationException: Get plan statistics did not complete successfully. StatusCode: 404, ReasonPhrase: 'Not Found', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Connection: close
  Date: Mon, 03 Jun 2019 12:53:30 GMT
  Server: Microsoft-HTTPAPI/2.0
  Content-Length: 315
  Content-Type: text/html; charset=us-ascii
}
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.Marketing.Automation.Data.AutomationPlanRepository.<GetAll>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Marketing.Automation.Client.Controllers.AutomationPlansController.<GetAll>d__8.MoveNext()

There was nothing in the Marketing Automation service’s logs to indicate an error at their end, and the IIS logs had this request on the CM server, but with a 500 status code rather than the 404 implied above.

Google didn’t help me, so after a bit of head scratching I raised a support ticket.

The solution

Having looked in more detail at all the config for my CM and Marketing Automation instances, support pointed out an obvious issue: The connection strings file on my CM server had the URL for Marketing Automation as an HTTP endpoint:

<?xml version="1.0" encoding="utf-8"?>
<connectionStrings>
  <!-- 
    Sitecore connection strings.
    All database connections for Sitecore are configured here.
  -->

  <add name="xdb.marketingautomation.operations.client" connectionString="http://xma.redacted.com" />
  <add name="xdb.marketingautomation.reporting.client" connectionString="http://xmareporting.redacted.com" />

</connectionStrings>

But the scripts which set up the services behind those endpoints only gave them HTTPS bindings – becasue xConnect services are secure-by-default. So that was clearly causing a failure when the CM instance tried to talk to the Marketing Automation services and got no response from the remote server, because it had no HTTP binding.

A bit of further digging lead to the embarrasing discovery that I’d managed to typo the Marketing Automation URL in the CM server SIF json file. Somehow I’d managed to use “http” instead of “https” there, and hence the CM server was set up to use the wrong URL.

So don’t be me. Proof-read your SIF scripts carefully before you deploy…

Pay attention to your index exclusions

Jeremy Davis — Thu, 04 Jul 2019 08:24:15 +0000

I hit an interesting issue recently: Some code that worked fine on a QA instance of Sitecore had been deployed for UAT and was now failing with an odd error message. Whilst this issue was entirely our fault, there wasn’t much in Google about the error messages I was seeing, so I’m trying to correct that problem today…

The issue

The code in question was runing on a Sitecore 7 instance. It fell over on UAT with this error:

[ArgumentNullException: Value cannot be null.Parameter name: key]
  System.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument) +49
  System.Collections.Generic.Dictionary`2.FindEntry(TKey key) +14545882
  System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value) +20
  Sitecore.ContentSearch.ContentSearchManager.GetIndex(String name) +52
  Sitecore.ContentSearch.ContentSearchManager.CreateSearchContext(IIndexable indexable) +17

That’s not a very helpful error, so I spent some time tracking it down. From the stack trace you can see that the problem seems to be in translating an IIndexable into a search context, but it doesn’t say anything helpful about why…

And under the surface, the code which was crashing looked something like:

var root = Sitecore.Context.GetItem("{08D9E0A1-BB72-48FD-AAB2-EFCD6F3B3C92}");
using (var context = ContentSearchManager.CreateSearchContext(new SitecoreIndexableItem(root)))
{
   // run some search queries starting from the "root" item...
}

The cause

A bit of digging with my old friend ILSpy lead to me working out that the exception being thrown here is because CreateSearchContext() above does two things:

public static IProviderSearchContext CreateSearchContext(IIndexable indexable)
{ 
  return GetIndex(indexable).CreateSearchContext(SearchSecurityOptions.EnableSecurityCheck);
}

It translates the IIndexable into a valid index for that item. And then it creates a search context on that index.

The exception gets thrown because the translation returns null – that item doesn’t map to any index. Under the surface, there’s a pipeline that gets run to do this translation, and it iterates all the defined indexes. And asks them in turn if they contain the item in question.

In the scenario I was looking at, all indexes returned “no” when asked this, and the code in the pipeline returns null when that happens… So later on when Sitecore tries to fetch its index by looking up the index based on a null name, you get the exception above.

Some further digging found a key difference between the configurations of the “working” servers and the “broken” one:

On the machines where it worked Sitecore’s search indexes were configured as “index all templates, except some specific ones”
But the servers that were broken were configured as “exclude all templates, except some specific ones”…

So the real cause of this issue was that the Sitecore Item passed into ContentSearchManager.CreateSearchContext() was based on a template that was specifically excluded from search indexing. As soon as I fixed this, the code started working.

Conclusions

Well, the obvious conclusions are: 1) Don’t exclude items you’re going to call ContentSearchManager.CreateSearchContext() on from your indexes. And 2) Manage your configuration across deployments better than this…

Having done some further testing, the error message above seems specific to older versions of Sitecore. I’ve tested this on V9.1 as well, and it still crashes in this scenario, but it shows a different exception:

[NullReferenceException: Object reference not set to an instance of an object.] 

 Sitecore.ContentSearch.SitecoreItemCrawler.IsExcludedFromIndex(SitecoreIndexableItem indexable, Boolean checkLocation) +69
  Sitecore.ContentSearch.SitecoreItemCrawler.GetContextIndexRanking(IIndexable indexable) +113
  System.Linq.WhereSelectListIterator`2.MoveNext() +116
  System.Linq.Enumerable.Min(IEnumerable`1 source) +72

 Sitecore.ContentSearch.AbstractSearchIndex.Sitecore.ContentSearch.Pipelines.GetContextIndex.IContextIndexRankable.GetContextIndexRanking(IIndexable indexable) +119
  Sitecore.ContentSearch.Pipelines.GetContextIndex.<>c\_\_DisplayClass6\_0.<RankContextIndexes>b\_\_0(ISearchIndex i) +71
  System.Linq.WhereSelectEnumerableIterator`2.MoveNext() +237
  System.Linq.Buffer`1..ctor(IEnumerable`1 source) +280
  System.Linq.<GetEnumerator>d__1.MoveNext() +115
  System.Linq.Buffer`1..ctor(IEnumerable`1 source) +280
  System.Linq.Enumerable.ToArray(IEnumerable`1 source) +89

 Sitecore.ContentSearch.Pipelines.GetContextIndex.FetchIndex.GetContextIndex(IIndexable indexable, GetContextIndexArgs args) +699

 Sitecore.ContentSearch.Pipelines.GetContextIndex.FetchIndex.Process(GetContextIndexArgs args) +48 (Object , Object ) +13
  Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args) +483
  Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists) +235
  Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain) +21
  Sitecore.Abstractions.CorePipelineWrapper.Run(String pipelineName, PipelineArgs args) +73

 Sitecore.ContentSearch.Pipelines.GetContextIndex.GetContextIndexPipeline.Run(ICorePipeline pipeline, GetContextIndexArgs args) +38

The trace is a bit more obvious – but for people new to Sitecore’s index configuration process that still might be confusing. So if you’re seeing either of the error messages shown above, check what you’re excluding from your indexes…

Logging generated passwords in SIF

Jeremy Davis — Mon, 24 Jun 2019 08:23:52 +0000

I’ve been looking at adjusting SIF scripts for a production deployment recently, and realised that sometimes you’d like SIF to generate random passwords for you, but you need them logged so you can reuse them in scripts you’re crafting for other roles. It doesn’t do that out of the box, but it turns out it’s actually quite simple:

The default configuration for “XPx-SingleDeveloper.json” includes a task that generates a set of random passwords (and some other stuff) for you:

    "Tasks": {
        "GeneratePasswords": {
            "Description": "Generates all shared passwords and secrets.",
            "Type": "SetVariable",
            "Params": {
                "Name": "XP1Passwords",
                "Scope": "Global",
                "Value":  [
                    /* Other generation tasks... */

                    {"SqlCorePassword":                "[variable('SqlCore.Password')]"},

                    /* Other generation tasks... */
                ]
            }
        }
    }

And if you’re feeling too lazy to generate all these things yourself, it seems sensible to reuse this in scripts where everything’s not on the same box…

It squirrels all these values away into variables, so that they can be reused for all the included tasks that set up all the roles – but it doesn’t seem to record them anywhere else. That’s entirely sensible from a security perspective – but that’s of no help if you’re going to have to take the Content Delivery .json over to another server and run it there. (Yes, I know SIF v2 supports remoting – but so far I’ve yet to meet a client that actually allows that on their production servers)

But it turns out that it’s actually very simple to get SIF to log something for you. There are two things to do:

First, you need to ensure that you’ve registered the “WriteInformation” task, which enables logging. Depending on what file you’re starting from, this might be done already – but what you need is to ensure that task is added to the “/Register/Tasks” block:

    "Register": {
        "Tasks": {
            "SetVariable": "Set-Variable",
            "WriteInformation": "Write-Information"
        }
    },

With that done, you can now add your own tasks to output whatever generated values you’re after. So after the generation block in “/Tasks” above, you might add:

    "Tasks": {

        /* Whatever generation tasks you need */

        "DisplaySqlCorePassword": {
            "Description": "Displays the Sql Core DB password.",
            "Type": "WriteInformation",
            "Params": {
                "MessageData": "[concat('Sql Core DB Password: ',variable('SqlCore.Password'))]",
                "InformationAction": "Continue"
            }
        }
    }

in order to get the password generated for the Core database. And you can add more of these to write out whatever passwords you need to reuse:

That’s just binding a task based on “WriteInformation” which outputs the string defined by “MessageData“. So you can construct more complex messages if you want, too.

Just remember not to leave log files lying around with these values in them…

A little PowerShell hack for sending files to a remote machine

Jeremy Davis — Mon, 10 Jun 2019 08:03:59 +0000

I was asked to do some configuration on a remote computer recently, and discovered that the security-concious network admins had locked down the ability for me to share my local computer’s files with the server via RDP and the ability to get to services like OneDrive. I had a collection of config files I had been asked to deploy, and manually creating each file on the server and copying over its contents seemed like a lot of hassle. So I tried a trick with PowerShell to make my life easier…

[Please note: I’m not condoning breaking network security policies here – don’t use this in a way that would get you into trouble!]

Ages ago I wrote a post about how you can embed files directly into PowerShell scripts using Base64 encoding. Since that idea can turn an arbitrary file into text that you can save in a script, it can also be used to encode an arbitrary file and place it onto the clipboard. And that gives you a route to move the file over an RDP session’s shared clipboard – which looked like it might be an approach to making my situation easier. So I hacked up a little PowerShell module that can be used to achieve this…

So on one machine we’re going to need to send a specific file to the clipboard. That entails a commandlet that takes the path to a file, checks it exists and sends the file to the clipboard:

function Send-ClipboardFile
{
    param(
        [string]$file
    )

    if(!(Test-Path $file))
    {
        throw "Specified file '$file' cannot be found"
    }

    $fileName = Split-Path $file -leaf

    $data = Base64Encode-File $file
    "$fileName`b$data" | Set-Clipboard

    Write-Host "File `"$file`" encoded to clipboard in $($data.Length) bytes"
}

Having checked the file exists, the code uses Split-Path to make sure it has just the file name, ignoring any relative path that may have been passed in. It then turns the file into a base64 string. Finally it sends a string to the clipboard that combines the file name and the encoded data and writes some output.

Encoding the file to a base64 string is pretty simple:

function Base64Encode-File
{
  param
  (
    [Parameter(Mandatory = $true)]
    [string]$file
  )
  process
  {
    $c = Get-Content $file -Encoding Byte 
    return [System.Convert]::ToBase64String($c)
  }
}

All it needs to do is read the contents of the file, and call the .Net function to convert the value into base64.

The other end of the process involves receiving the data from the clipboard:

function Receive-ClipboardFile
{
    param(
        [string]$alternativeName = $null
    )

    $clip = Get-Clipboard

    $bits = $clip.Split("`b")

    if($bits.Length -ne 2)
    {
        throw "Data error. Expected 2 items of data on clipboard, but got $($bits.Length)"
    }

    $file = $bits[0]
    if($alternativeName -ne $null -and $alternativeName.Length -gt 0)
    {
        $file = $alternativeName
    }
    $data = $bits[1]

    Write-Host "Receiving file `"$file`" from $($clip.Length) bytes of data"

    Write-EmbeddedFile $data $file
}

This bit of code grabs the data that’s on the clipboard, and tries to split it using the separator character that was used in the sending routine above. It checks to see that it got exactly two items from that. If not, that implies an error. Otherwise you have a filename and a block of encoded data. If the user passed the optional alternative file name parameter then it uses that, otherwise it uses the original file name sent over the clipboard. Then it can write some output, and then turn the clipboard data back into a disk file.

And again, writing the data out to disk is pretty simple:

function Write-EmbeddedFile
{
  param
  (
    [string]$base64,
    [string]$targetFile
  )
  process
  {
    $Content = [System.Convert]::FromBase64String($base64)
    Set-Content -Path ".\$targetFile" -Value $Content -Encoding Byte
  }
}

Finally, the module file can export the two key functions for use:

Export-ModuleMember -Function "Receive-ClipboardFile"
Export-ModuleMember -Function "Send-ClipboardFile"

And with that, all you have to do is drop this script (I named it “ClipboardFileTransfer.psm1“) into a suitably named folder in your modules directory or locally and you can import it and send some files…

For example, if you have the module in a local directory on the local machine you might run:

And on the remote one you can run:

And magically the file is transferred… (And of course you can do it in the opposite direction too)

[Note to self: tweak script so that the bytes reported at both ends measure the same thing… ]

You need to get this one script onto both of your machines, but if you can do that, this allows for easy transfers of other files. And it made my day a lot simpler.

If you think this might be useful, the script is available as a gist for you to use.

A pain point with “Trusted Connection” in Sitecore v9.1

Jeremy Davis — Mon, 27 May 2019 07:26:02 +0000

One of the projects I’m working on at the moment came with a requirement to change Sitecore v9.1 from running with the default SQL Security accounts to trusted connections using specific Active Directory accounts that the client provided. While there’s a bit of work to do to enable this, it shouldn’t be too tough. But trying to be a bit clever, I hit upon an issue which seemed worth documenting…

A bit of background:

You may well never have delved into the SQL Databases that Sitecore provisions, but some of the more modern ones include a stored procedure called GrantLeastPrivilege. They don’t seem to be officially documented, but when you examine them, they’re helper scripts that grant exactly the database rights that a user account needs to be able to access a specific Sitecore database:

So you’d hope that they’d be helpful if you need to move from one set of database accounts to another…

The issue:

But when I tried to use these scripts with the Active Directory accounts provided by the client, I got oddly inconsistent results. When I tried to call the procedure in some databases with my user

USE [MyInstance\_Xdb.Collection.Shard0]
GO
[xdb\_collection].[GrantLeastPrivilege] 'Domain\ProcessAccountName'

I got success. It would write out all the changes it made. But when I called the same procedure in other databases, with the same approach

USE [MyInstance\_ProcessingEngineStorage]
GO
[sitecore\_processing\_storage].[GrantLeastPrivilege] 'Domain\ProcessAccountName'

I got error messages instead – saying something like:

Msg 102, Level 15, State 1, Line 9Incorrect syntax near '\'.

When I tested it, across all the non-content databases, it broke on ProcessingEngineStorage, MarketingAutomation, Processing.Pools, Processing.Tasks, ProcessingEngineTasks and ReferenceData.

Looking at the underlying code, the scripts which work run statements in the form of

EXECUTE('grant execute on TYPE::[database].[thing] TO [' + @Name + ']')

and the ones that fail look like

EXECUTE('grant execute on TYPE::[database].[thing] TO ' + @Name)

so it’s pretty easy to tweak them to be successful. But it’s a bit of a pain that this doesn’t work out of the box. So I’ve reported this to Sitecore Support, who have accepted it’s a bug. If you’re hitting this issue too, make reference to bug 331327.

(And yes, I’m aware of the whole “string concatenation is dangerous when generating SQL statements thing – that’s a debate for another day)

Dealing with .tar.gz files on Windows Server

Jeremy Davis — Mon, 13 May 2019 07:17:55 +0000

A couple of times recently, I’ve found myself needing to deploy files that come wrapped in a .tar.gz archive onto servers. On your desktop that’s not too much of a problem – you just run the installer for your preferred 3rd party tool, or maybe use the new Unixy shell and you get on with it. But on client servers security can be higher and you don’t always get the option to run any old installer. So I needed an alternative…

Looking for inspiration, I did a bit of googling and came across a thread on Stack Overflow which suggested there is a PowerShell extension for handling .tar files. It’s based on 7Zip’s libraries – but it doesn’t require installing the full 7Zip toolset, and it can be fetched direct from the PowerShell module feed.

That seemed like a good staring point – but the code in the answers was going to need a bit of work. So I’ve taken that as a basis and produced my own script to use to deal with .tar.gz archives.

The main script

To be useful, the script is going to need to receive a file to extract, and a folder to put the results into. That’s easily done by declaring a couple of mandatory parameters:

[cmdletbinding(SupportsShouldProcess=$True)]
param(
    [Parameter(Mandatory=$True)]
    [string]$FileToExtract,

    [Parameter(Mandatory=$True)]
    [string]$TargetFolder,

    [int]$BufferSize = 1024
)

The parameters also declare that this script will honour the “-whatif” parameter via the cmdletbinding() attribute at the top. And it declares an optional parameter for the size of the file IO buffer used when extracting the .gzip stream – more on that later.

The logic of the script is fairly simple. First it needs to do some basic validation of the file it’s going to process. First it can test the file actually exists:

if(!(Test-Path $FileToExtract))
{
    throw "Source file '$FileToExtract' does not exist"
}

And then it can test that it has the right extension:

if(!$FileToExtract.EndsWith(".tar.gz", "CurrentCultureIgnoreCase"))
{
    throw "Source file '$FileToExtract' does not have a .tar.gz extension"
}

Once it’s happy, the overall processing is broken up into six operations. The first two make sure that the source file has an absolute path, before working out the right name for the .tar file that will be hiding inside the .tar.gz file.

Once that’s done, the real work is to expand the GZip data to get the .tar file. That can be done in native .Net code that I’ll get to in a sec. Then it has to make sure the extension for handling .tar files is installed – by grabbing a copy of the “7Zip4PowerShell” PowerShell module if it’s not available already. That can then be called to extract the data, before finally deleting the temporary .tar file:

$FileToExtract = Resolve-Path $FileToExtract
$tarFile = Calculate-TarFileName $FileToExtract

Expand-GZip $FileToExtract $tarFile $BufferSize
Ensure-7Zip
Extract-Tar $tarFile $TargetFolder

if ($PSCmdlet.ShouldProcess($tarFile,'Remove temporary tar file')) {
    Remove-Item $tarFile
}

What’s in all those functions? Read on…

Expanding the GZip stuff

The first step is working out the name for the output from the GZip file – which is the tar file. That’s pretty trivial, as it just means stripping the final “.gz” off the input filename:

function Calculate-TarFileName {
    param(
        [Parameter(Mandatory=$true)]
        [string] $targzFile
    )

    $targzFile.Substring(0, $targzFile.LastIndexOfAny('.'))
}

Expanding the GZip file takes a bit more work. This is largely cribbed from one of the answers on the Stack Overflow thread referenced above – bit with some enhancements. Firstly it now understands -whatif and won’t actually generate the new file in that scenario. And secondly it adds some code to enable a progress bar. Other than that, it’s basically just processing the GzipStream using standard .Net code:

function Expand-GZip {
    [cmdletbinding(SupportsShouldProcess=$True)]
    param(
        [Parameter(Mandatory=$true)]
        [string]$infile,
        [Parameter(Mandatory=$true)]
        [string]$outFile,
        [int]$bufferSize = 1024
    )
    $fileSize = Original-GzipFileSize $inFile
    $processed = 0

    if ($PSCmdlet.ShouldProcess($infile,"Expand gzip stream")) {
        $input = New-Object System.IO.FileStream $inFile, ([IO.FileMode]::Open), ([IO.FileAccess]::Read), ([IO.FileShare]::Read)
        $output = New-Object System.IO.FileStream $outFile, ([IO.FileMode]::Create), ([IO.FileAccess]::Write), ([IO.FileShare]::None)
        $gzipStream = New-Object System.IO.Compression.GzipStream $input, ([IO.Compression.CompressionMode]::Decompress)

        $buffer = New-Object byte[]($bufferSize)
        while($true){

            $pc = (($processed / $fileSize) * 100) % 100
            Write-Progress "Extracting tar from gzip" -PercentComplete $pc

            $read = $gzipstream.Read($buffer, 0, $bufferSize)

            $processed = $processed + $read

            if ($read -le 0)
            {
                Write-Progress "Extracting tar from gzip" -Completed
                break
            }
            $output.Write($buffer, 0, $read)
        }

        $gzipStream.Close()
        $output.Close()
        $input.Close()
    }
}

Making a useful progress bar involves knowing the size of the final stream, however. But Google to the rescue again here, as it pointed me towards this CodeProject posting that describes the C# code to achieve this. Turns out you just need to verify that it’s actually a GZip stream (looking at the first three bytes) and the find the last four bytes to get an Int32 that is the original length:

function Original-GzipFileSize {
    param(
        [Parameter(Mandatory=$true)]
        [string] $gzipFile
    )

    $fs = New-Object System.IO.FileStream $gzipFile, ([IO.FileMode]::Open), ([IO.FileAccess]::Read), ([IO.FileShare]::Read)

    try
    {
        $fh = New-Object byte[](3)
        $fs.Read($fh, 0, 3) | Out-Null
        # If magic numbers are 31 and 139 and the deflation id is 8 then this is a file to process
        if ($fh[0] -eq 31 -and $fh[1] -eq 139 -and $fh[2] -eq 8) 
        {
            $ba = New-Object byte[](4)
            $fs.Seek(-4, [System.IO.SeekOrigin]::End) | Out-Null
            $fs.Read($ba, 0, 4) | Out-Null

            return [int32][System.BitConverter]::ToInt32($ba, 0)
        }
        else
        {
            throw "File '$gzipFile' does not have the correct gzip header"
        }
    }
    finally
    {
        $fs.Close()
    }
}

So the Expand-GZip function can use that to work out a percentage completion as it iterates through blocks of the stream…

I mentioned before that the overall script has an option for block sizes for processing here. It defaults to 1KB here because that’s what was in the code I cribbed, but you can pass a bigger block size to trade off speed for memory usage.

But once that’s complete the initial .tar.gz file will have a .tar alongside it.

Dealing with the .tar file

The original Stack Overflow thread included an answer that suggested the “7Zip4PowerShell” module for PowerShell would be the simplest approach here. This code breaks it up into two tasks – one to make sure the module is available locally to use, and the other to actually use it.

The thread talks about two approches to using that module. One where you manually copy the required files locally and the script picks those up and uses them, and one where it asks the Install-Package commandlet to fetch it from the public feed. For laughs I decided to combine the two, as I could see scenarios where both mught be useful.

So if you put the files for the package to the “7Zip4Powershell” folder next to the script, it’ll spot them and use this local copy. To get those files you can run

Save-Module -Name 7Zip4Powershell -Path .

from a prompt. This approach will be most useful on highly-secured servers where the admins want to vet all the files your work uses before you do anything with them. You can hand over both this script and the files for that module for investigation, and they just need copying to the server for use.

Alternatively, if that folder does not exist, it’ll pull the module from the public feed. That’s the “zero effort” approach when you’re allowed to use it. You don’t need to do any extra work – it’ll just pull in the right code if it’s not already installed for you. And as before, the logic is wrapped up to support -WhatIf.

function Ensure-7Zip {
    param(
        [string]$pathToModule = ".\7Zip4Powershell\1.9.0\7Zip4PowerShell.psd1"
    )

    if (-not (Get-Command Expand-7Zip -ErrorAction Ignore)) {
        if(Test-Path $pathToModule)
        {
            if ($PSCmdlet.ShouldProcess($pathToModule,"Install 7Zip module from local path")) {
                Write-Progress -Activity "Installing the 7Zip4PowerShell module" "Using local module" -PercentComplete 50
                Import-Module $pathToModule
                Write-Progress -Activity "Installing the 7Zip4PowerShell module" "Using local module" -Completed
            }
        }
        else
        {
            if ($PSCmdlet.ShouldProcess("PowerShell feed",'Install 7Zip module')) {
                Write-Progress  -Activity "Installing the 7Zip4PowerShell module" "Using public feed" -PercentComplete 50
                $progressPreference = 'silentlyContinue'
                Install-Package -Scope CurrentUser -Force 7Zip4PowerShell > $null
                $progressPreference = 'Continue'
                Write-Progress  -Activity "Installing the 7Zip4PowerShell module" "Using public feed" -Completed
            }
        }
    }
}

At the moment this code is hard-coded to the current version of this module – Making that more flexible is on my backlog, if I get a chance…

Once the code installed, the actual command to run the extraction is easy:

function Extract-Tar {
    [cmdletbinding(SupportsShouldProcess=$True)]
    param(
        [Parameter(Mandatory=$true)]
        [string] $tarFile,
        [Parameter(Mandatory=$true)]
        [string] $dest
    )

    if ($PSCmdlet.ShouldProcess($tarFile,"Expand tar file")) {
        Expand-7Zip $tarFile $dest
    }
}

All that adds on top of the 7Zip command is the “-WhatIf” logic, as that doesn’t seem to be supported by default.

In action…

So with that installed, you can extract files to your hearts content:

And if you pass -WhatIf then it tells you what it would do, but does nothing:

I’ve put the full code up in a gist if you think it would be useful for you…

Pasting multiple cells into Excel from PowerShell

Jeremy Davis — Mon, 15 Apr 2019 07:31:34 +0000

Sometimes the learning point from working on a misbehaving Sitecore server isn’t related to the CMS. Recently I learned something useful about Excel while I was addressing some other issues. Not sure if this is “so trivially simple I’m just the last one to realise” or whether it’s a really useful bit of trivia – but just because someone else might benefit:

I was monitoring an issue with a Sitecore server recently, and to keep track of what had been going on, I’d knocked up a graph in Excel. Every so often I was looking at a certain data value on the server, and then copy/pasting that into the spreadsheet along with the date/time of when the measurement was taken.

Having done that a few times I was struck by the urge to make my life simpler. For… reasons… it wasn’t practical to have PowerShell writing out a CSV file to import into Excel in one go – so could I get PowerShell to give me the date and the value on my clipboard, so I’d just have to paste it into my spreadsheet?

Yes of course:

First up, getting the current date/time is handled by the “Get-Date” commandlet. It happened that the server I was using was configured for a US locale, and being a Brit that meant I wanted my date formatted differently to the default output. Not a problem – you just need to tell PowerShell how you’d like your date using standard .Net format string data:

$timeStamp = Get-Date -Format "dd/MM/yyyy hh:mm"

Grabbing the actual measurement isn’t relevant here, but I squirrelled that data away in a variable too:

$value = /\* someMagicToMeasureAValue \*/

And you can send stuff to the clipboard automatically with the “Set-Clipboard” commandlet.

But my spreadsheet was formatted with a column for the timestamp and a column for the value, so I could plot a graph from the data. So how do you get the data onto the clipboard in the right format so that pasting into Excel will make sure it goes into two cells?

Well after some fun with Google, it turns out this is kind of obvious in retrospect: You just tab-separate your data. In PowerShell “`t” (back-tick, followed by “t”) is a tab character. And that means I can send my two columns of data to the clipboard with:

"$timestamp `t $value" | Set-Clipboard

And if I hit paste into Excel, I get:

And you can put multiple tabs in if you want to paste more than one column…