loading...
Xpirit

What to do when your build hangs on the Hosted Pool...

jessehouwing profile image Jesse Houwing Originally published at jessehouwing.net on ・3 min read

What to do when your build hangs on the Hosted Pool...

Both GitHub actions and Azure Pipelines offer the ability tor un your CI pipeline in the cloud. These agents are provisioned in Azure and fully maintained by Microsoft. These hosted runners are cost effective, require no maintenance and are basically free to use in many cases.

But what to do when your build just freezes and stops responding?

If this were your own runner, or if it's hosted by your company, you could probably remote into the machine to see what's going on. Unfortunately, that's no option with the hosted runners.

After cancelling the build you could inspect the logs for any hints, but they may not reveal anything useful either.

Now what?!

What I really wanted, was to have a quick peek at the desktop of the agent. A screenshot would do... So that is what I set out to accomplish. A quick google gave me 9 different command-line tools to grab a screenshot of the desktop and one stood out for its ease of use screenshot-cmd. It's a simple portable executable that can be downloaded directly. So I cobbled up a little PowerShell to download and run it and add the screenshot to the logs:

Invoke-WebRequest -Uri "https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/screenshot-cmd/screenshot-cmd.exe" -OutFile "$(Agent.ToolsDirectory)\screenshot-cmd.exe"

& "$(Agent.ToolsDirectory)\screenshot-cmd.exe"

Write-Host "##vso[task.uploadfile]$(Agent.ToolsDirectory)\screenshot.png"

and got a screenshot in the logs:

What to do when your build hangs on the Hosted Pool...

Telling me... nothing...

What to do when your build hangs on the Hosted Pool...
A screenshot of the PowerShell console my little tool was launched from.

And some more StackOverflow hunting lead me to a snippet to minimize that console.

Add-Type -Name ConsoleUtils -Namespace WPIA -MemberDefinition @'
      [DllImport("Kernel32.dll")]
      public static extern IntPtr GetConsoleWindow();
      [DllImport("user32.dll")]
      public static extern bool ShowWindow(IntPtr hWnd, Int32 nCmdShow);
   '@

$ConsoleMode = @{
    HIDDEN = 0;
    NORMAL = 1;
    MINIMIZED = 2;
    MAXIMIZED = 3;
    SHOW = 5
    RESTORE = 9
}

$hWnd = [WPIA.ConsoleUtils]::GetConsoleWindow()
[WPIA.ConsoleUtils]::ShowWindow($hWnd, $ConsoleMode.MINIMIZED)

Which finally revealed:

What to do when your build hangs on the Hosted Pool...
The Just-in-Time Debugger is still enabled on the Visual studio 2017 agent.

Disabling the Just-in-time debugger is a simple matter of resetting a couple of registry keys. And now the build no longer freezes and is telling me what's wrong.

& reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug" /v Debugger /d - /t REG_SZ /f
& reg add "HKLM\SOFTWARE\Microsoft\.NETFramework" /v DbgManagedDebugger /d - /t REG_SZ /f
& reg add "HKCU\Software\Microsoft\Windows\Windows Error Reporting" /v DontShowUI /d 1 /t REG_DWORD /f

Bliss!

Putting it all together

The final script looks like this (in YAML):

steps:
- powershell: |
   Add-Type -Name ConsoleUtils -Namespace WPIA -MemberDefinition @'
      [DllImport("Kernel32.dll")]
      public static extern IntPtr GetConsoleWindow();
      [DllImport("user32.dll")]
      public static extern bool ShowWindow(IntPtr hWnd, Int32 nCmdShow);
   '@

   $ConsoleMode = @{
    HIDDEN = 0;
    NORMAL = 1;
    MINIMIZED = 2;
    MAXIMIZED = 3;
    SHOW = 5
    RESTORE = 9
    }

   $hWnd = [WPIA.ConsoleUtils]::GetConsoleWindow()

   $a = [WPIA.ConsoleUtils]::ShowWindow($hWnd, $ConsoleMode.MINIMIZED)

   Invoke-WebRequest -Uri "https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/screenshot-cmd/screenshot-cmd.exe" -OutFile "$(Agent.ToolsDirectory)\screenshot-cmd.exe"
   & "$(Agent.ToolsDirectory)\screenshot-cmd.exe"
   Write-Host "##vso[task.uploadfile]$(Agent.ToolsDirectory)\screenshot.png"
  workingDirectory: '$(Agent.ToolsDirectory)'
  displayName: 'PowerShell Script'
  condition: always()

The condition: always() ensures that the screenshot is taken after a request to cancel the build has been sent.

And this is the UI based equivalent:

What to do when your build hangs on the Hosted Pool...

What do you think?

Is this valuable enough to turn into a custom task and publish it to the marketplace?

Posted on by:

jessehouwing profile

Jesse Houwing

@jessehouwing

I love good coffee, meaning strong, dark espresso! Regular Expressions are like word-puzzles to me, as are LINQ queries. I take pictures everywhere I go. Husband of 1, father of 2.

Xpirit

Xpirit helps you solve your IT problems. We act in multiple roles, from boardroom advisor, coding architect and lead developer to trainer and coach.

Discussion

markdown guide