DEV Community

Haris Ali Khan
Haris Ali Khan

Posted on • Originally published at Medium

Multiple VMs, One Server, Zero Bottlenecks

#vpsperformancebottleneck #1pcmultiplevm #win2025vmsetupchallanges #vpsperformanceissue

Multiple VMs, One Server, Zero Bottlenecks: A Complete Performance Guide

The Setup That Looks Fine on Paper

You have a powerful dual-socket server. Plenty of cores (32, 64, or more logical processors across two physical CPUs). Plenty of RAM. You spin up two or three VMs, each running its own independent stack. Clean, economical, manageable.

Everything works. Until it doesn't.

Under load, VM performance becomes unpredictable. One VM spikes and the others slow down. Process launch times balloon. Response times suffer. The hardware is powerful but something invisible is causing the VMs to strangle each other.

What We Tried First, and What Did Not Work

When VM performance degrades on a well-specced server, these are the things that immediately come to mind. We tried most of them. None solved the core problem.

Increasing vCPU allocation in Hyper-V. Gave each VM more virtual processors. Performance barely changed. vCPUs are not physical cores, and more of them increases scheduling complexity rather than throughput.

Increasing RAM allocation. Already sufficient. Memory was not the constraint. Adding more made no difference to latency.

Adjusting VM CPU reserve and weight settings. Hyper-V allows you to set CPU reserve percentages and relative weights between VMs. These are soft controls that influence priority but do not isolate workloads. Under full load, VMs still competed on the same physical cores.

High Performance power plan. Applied to host and VMs. Helped marginally with CPU frequency consistency but did not address the scheduling contention.

Disk optimisation: SSD vs HDD, partition alignment, defragmentation. Our workload was not I/O bound. Storage was already on SSD RAID. Partition alignment was correct. Defragmentation on SSDs is irrelevant. This was a red herring.

Display adapter and remote access optimisation. Switched from the default Microsoft Hyper-V Video adapter (no GPU acceleration) to HvSocket for enhanced sessions. Improved remote desktop responsiveness but had zero effect on workload performance.

Disabling background services. Turned off Windows Search, SysMain, Windows Update, and similar services on VMs. Reduced idle noise but did not address the root cause under load.

Dynamic Memory. Tried both enabling and disabling Hyper-V Dynamic Memory. Neither configuration affected CPU scheduling behaviour.

None of these addressed what was actually happening: all virtual processors from all VMs were competing for the same pool of physical logical processors with no isolation boundary between them.

The Actual Problem: How Shared CPU Hurts VMs

When Hyper-V runs multiple VMs on a single host, all virtual processors compete for the same pool of logical processors by default. The hypervisor scheduler has no concept of your workloads being independent or performance-critical. It time-slices everything together.

The consequences are invisible until you measure them:

Cache eviction. When VM A is scheduled onto a core that VM B was just using, VM B's cache data gets displaced. VM B then has to reload everything from RAM. At scale this adds measurable latency to every operation.

NUMA cross-traffic. A dual-socket server has two separate memory controllers, one per CPU socket. When a VM's virtual processor runs on Socket 1 but its memory sits on Socket 2, every memory access crosses the NUMA interconnect. This is slower than local access and compounds under load.

Scheduling jitter. The hypervisor is constantly making scheduling decisions across all logical processors simultaneously. A VM waiting for a physical core to become available shows up as vCPU wait time. This is the most direct measure of contention.

The root problem is that the hypervisor treats all logical processors as a flat pool. When any workload runs hot, the others pay the price. The fix is to stop sharing. Give each workload its own physical processors and enforce that boundary at the hypervisor level.

Measuring the Problem

Before changing anything, establish a baseline. You need numbers to compare against later.

Process launch time is the most universal proxy for scheduling health. On Windows, launching any process requires CPU time from the scheduler. If the scheduler is under pressure, process launch inflates immediately.

# Run on each VM
$times = 1..10 | ForEach-Object {
  $sw = [System.Diagnostics.Stopwatch]::StartNew()
  $p = Start-Process cmd -ArgumentList "/c exit" -PassThru -WindowStyle Hidden
  $p.WaitForExit()
  $sw.Stop()
  $sw.Elapsed.TotalMilliseconds
}
$avg = [math]::Round(($times | Measure-Object -Average).Average, 0)
Write-Host "Average process launch: $avg ms"
Enter fullscreen mode Exit fullscreen mode

A healthy VM on modern hardware should launch a process in 50–80ms. Above 150ms indicates a scheduling problem. Above 250ms is severe.

vCPU Wait Time Per Dispatch shows exactly how long virtual processors wait for a physical core. Run this on the Hyper-V host:

Get-Counter "\Hyper-V Hypervisor Virtual Processor(*)\CPU Wait Time Per Dispatch" `
  -SampleInterval 5 -MaxSamples 3 |
  Select-Object -ExpandProperty CounterSamples |
  Select-Object InstanceName, @{N='WaitNs';E={[math]::Round($_.CookedValue,0)}} |
  Sort-Object WaitNs -Descending | Select-Object -First 10
Enter fullscreen mode Exit fullscreen mode

Below 5,000 ns is healthy. Above 20,000 ns is contention. Above 100,000 ns is a serious problem.

Record both before making any changes.

The Two-Socket Advantage

A dual-socket server is not just a server with more cores. It is two physically separate CPUs sharing a motherboard, each with its own memory controller and its own set of logical processors. This is the NUMA architecture.

On a typical dual-socket system:

  • Socket 1 owns LP 0 through LP (N/2 − 1)
  • Socket 2 owns LP N/2 through LP (N − 1)

On a 64-LP system: Socket 1 covers LP 0 through 31, Socket 2 covers LP 32 through 63.

This boundary is the key to isolation. If you confine the host OS workload to Socket 1 and all VMs to Socket 2, or split VMs deliberately across sockets, you eliminate cross-socket scheduling interference entirely. The two sockets stop competing. Each runs independently.

To read your own topology, run this on the Hyper-V host after obtaining CpuGroups.exe:

CpuGroups.exe GetCpuTopology
Enter fullscreen mode Exit fullscreen mode

This shows every LP with its NUMA node, package ID, and core ID. Identify where your socket boundary is.

The Tool: CpuGroups.exe

Hyper-V has a mechanism called CPU Groups that allows you to create logical partitions of the host's physical processors and assign VMs exclusively to those partitions. This is what makes socket isolation possible.

The tool is CpuGroups.exe, available from the Microsoft Download Center. It is not built into Windows. Download and run it as an administrator on the Hyper-V host.

Key commands:

CpuGroups.exe GetCpuTopology     # Read LP layout
CpuGroups.exe CreateGroup /GroupId:<GUID> /GroupAffinity:<LP,LP,...>
CpuGroups.exe SetVmGroup /VmName:<n> /GroupId:<GUID>  # VM must be stopped
CpuGroups.exe GetVmGroup       # Verify assignments
CpuGroups.exe GetGroups        # See all groups
Enter fullscreen mode Exit fullscreen mode

Obstacle 1: VBS Blocks CPU Group Creation on Windows Server 2025

The first time you run CreateGroup you will likely see:

Failed with error 0xc0350005
Enter fullscreen mode Exit fullscreen mode

0xc0350005 is ERROR_HV_INVALID_PARAMETER. On Windows Server 2025 this is typically caused by Virtualization Based Security (VBS) being active. VBS uses the hypervisor to create an isolated security environment, and CPU group creation is one of the things it restricts.

Check VBS status:

$vbs = Get-CimInstance -ClassName Win32_DeviceGuard -Namespace root\Microsoft\Windows\DeviceGuard
Write-Host "VBS Status: $($vbs.VirtualizationBasedSecurityStatus)"
# 0 = disabled, 2 = running
Enter fullscreen mode Exit fullscreen mode

To disable VBS without breaking Hyper-V, apply these registry keys. On Windows Server 2025, PowerShell's New-Item may fail silently due to kernel protection. Use reg.exe via cmd /c for the policy key instead:

# Main DeviceGuard key
$dg = "HKLM:\SYSTEM\CurrentControlSet\Control\DeviceGuard"
Set-ItemProperty -Path $dg -Name "EnableVirtualizationBasedSecurity" -Value 2 -Type DWord
Set-ItemProperty -Path $dg -Name "Unlocked"             -Value 1 -Type DWord
Set-ItemProperty -Path $dg -Name "Locked"              -Value 0 -Type DWord
Set-ItemProperty -Path $dg -Name "LsaCfgFlags"            -Value 0 -Type DWord

# Policy key — use reg.exe on WS2025
cmd /c "reg add ""HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard"" /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 2 /f"
cmd /c "reg add ""HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard"" /v HypervisorEnforcedCodeIntegrity /t REG_DWORD /d 0 /f"
cmd /c "reg add ""HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard"" /v LsaCfgFlags /t REG_DWORD /d 0 /f"
Enter fullscreen mode Exit fullscreen mode

Reboot. Verify VBS status shows 0 before proceeding.

Disabling VBS removes Credential Guard and HVCI protections. Evaluate this against your security requirements. On dedicated, isolated infrastructure running trusted workloads this is generally acceptable.

Obstacle 2: The Hypervisor Scheduler

After disabling VBS and rebooting, CpuGroups.exe may still return 0xc0350005. The second obstacle is the hypervisor scheduler type.

Windows Server 2025 defaults to the Core Scheduler (0x3), introduced to mitigate Spectre/Meltdown-class vulnerabilities on SMT-enabled systems. The Core scheduler is incompatible with CPU group creation.

Check the active scheduler:

Get-WinEvent -FilterHashTable @{ProviderName="Microsoft-Windows-Hyper-V-Hypervisor"; ID=2} -MaxEvents 1 |
  Select-Object -ExpandProperty Message
# "Hypervisor scheduler type is 0x3" = Core | "0x2" = Classic
Enter fullscreen mode Exit fullscreen mode

Switch to the Classic Scheduler:

bcdedit /set hypervisorschedulertype classic
Enter fullscreen mode Exit fullscreen mode

Verify:

bcdedit /enum | Select-String "scheduler"
# hypervisorschedulertype  Classic
Enter fullscreen mode Exit fullscreen mode

Reboot. After this reboot, CpuGroups.exe CreateGroup will succeed.

The Classic scheduler uses fair-share round-robin scheduling and does not provide the SMT security boundary of the Core scheduler. On servers running trusted workloads in a controlled environment this is generally acceptable.

The Complete Step-by-Step

Step 1: Baseline
Measure process launch time and vCPU wait on host and all VMs.

Step 2: Disable VBS
Apply the DeviceGuard registry keys above.

Step 3: Switch Scheduler

bcdedit /set hypervisorschedulertype classic
Enter fullscreen mode Exit fullscreen mode

Step 4: Reboot
One reboot handles both.

Step 5: Verify

# VBS should be 0
(Get-CimInstance Win32_DeviceGuard -Namespace root\Microsoft\Windows\DeviceGuard).VirtualizationBasedSecurityStatus

# Scheduler should be 0x2
Get-WinEvent -FilterHashTable @{ProviderName="Microsoft-Windows-Hyper-V-Hypervisor"; ID=2} -MaxEvents 1 | Select-Object -ExpandProperty Message
Enter fullscreen mode Exit fullscreen mode

Step 6: Read Topology

CpuGroups.exe GetCpuTopology
Enter fullscreen mode Exit fullscreen mode

Identify your socket LP ranges.

Step 7: Create CPU Group

CpuGroups.exe CreateGroup /GroupId:a5cf3c7e-6033-49c3-84cb-000000000001 /GroupAffinity:32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63
Enter fullscreen mode Exit fullscreen mode

Adjust the LP range to match your Socket 2 topology.

Step 8: Assign VMs

Stop-VM "VM-1" -Force
Stop-VM "VM-2" -Force
Start-Sleep 20
Enter fullscreen mode Exit fullscreen mode 
CpuGroups.exe SetVmGroup /VmName:"VM-1" /GroupId:a5cf3c7e-6033-49c3-84cb-000000000001
CpuGroups.exe SetVmGroup /VmName:"VM-2" /GroupId:a5cf3c7e-6033-49c3-84cb-000000000001
CpuGroups.exe GetVmGroup
Enter fullscreen mode Exit fullscreen mode 
Start-VM "VM-1"
Start-Sleep 45
Start-VM "VM-2"
Enter fullscreen mode Exit fullscreen mode

Step 9: Verify Socket Separation

Get-Counter "\Hyper-V Hypervisor Logical Processor(*)\% Total Run Time" -SampleInterval 3 -MaxSamples 1 |
  Select-Object -ExpandProperty CounterSamples |
  Where-Object { $_.InstanceName -ne "_total" } |
  Select-Object @{N='LP';E={[int]($_.InstanceName -replace 'hv lp ','')}},
         @{N='Socket';E={if([int]($_.InstanceName -replace 'hv lp ','') -lt 32){'S1-Host'}else{'S2-VMs'}}},
         @{N='Pct';E={[math]::Round($_.CookedValue,0)}} |
  Sort-Object LP | Format-Table -AutoSize
Enter fullscreen mode Exit fullscreen mode

You should see complete separation, Socket 1 LPs carrying host load, Socket 2 LPs carrying VM load, zero crossover.

Making It Survive Reboots

CPU groups do not persist across reboots. When the host reboots, all group assignments are gone and VMs drift back to competing on all LPs. A startup script registered as a scheduled task handles this automatically:

@echo off
REM RestoreCpuGroup.bat
timeout /t 60 /nobreak > nul
C:\Tools\CpuGroups.exe GetGroups | findstr /i "a5cf3c7e" > nul
if errorlevel 1 (
  C:\Tools\CpuGroups.exe CreateGroup /GroupId:a5cf3c7e-6033-49c3-84cb-000000000001 /GroupAffinity:32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63
  C:\Tools\CpuGroups.exe SetVmGroup /VmName:"VM-1" /GroupId:a5cf3c7e-6033-49c3-84cb-000000000001
  C:\Tools\CpuGroups.exe SetVmGroup /VmName:"VM-2" /GroupId:a5cf3c7e-6033-49c3-84cb-000000000001
)
Enter fullscreen mode Exit fullscreen mode 
$action  = New-ScheduledTaskAction -Execute "C:\Tools\RestoreCpuGroup.bat"
$trigger  = New-ScheduledTaskTrigger -AtStartup
$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 5)
$principal = New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest
Register-ScheduledTask -TaskName "RestoreCpuGroup" -Action $action -Trigger $trigger `
  -Settings $settings -Principal $principal -Force
Enter fullscreen mode Exit fullscreen mode

Results

After the full optimisation on our dual Intel E5-2698 v3 system (64 LPs total, three independent workloads):

Metric Before After
Process launch, VM 1 ~250 ms ~62 ms
Process launch, VM 2 ~242 ms ~53 ms
vCPU wait, VM 1 ~225,000 ns ~35,000 ns
vCPU wait, VM 2 ~374,000 ns ~18,000 ns

Under full simultaneous load, with the host running hard on Socket 1 and both VMs processing independently on Socket 2, CPU consumption on each socket was entirely independent. A spike on Socket 1 produced zero measurable impact on Socket 2. The LP utilisation counters showed clean separation with no crossover.

Reference, System Specifications

The following configuration was used during this work. Included for readers replicating this in similar environments.

Host Hardware

  • Server: Custom-built rackmount (Coretron 2)
  • CPU: 2× Intel Xeon E5-2698 v3 @ 2.30 GHz (16 cores / 32 threads each, 32 cores / 64 logical processors total)
  • RAM: 256 GB DDR4 ECC Registered
  • Storage: SSD RAID array (OS and VM virtual disks on SSD)
  • NUMA Nodes: 2 (Socket 1: LP 0–31, Socket 2: LP 32–63)

Host OS

  • Windows Server 2025 Datacenter (Evaluation)
  • Hyper-V Role enabled
  • Hypervisor Scheduler: Classic (0x2), changed as part of this guide
  • VBS: Disabled, changed as part of this guide
  • NUMA Spanning: Enabled (required, VMs exceed single-node RAM capacity at 64 GB each)

Virtual Machines

  • VM 1 (NITRON 3.0): Windows Server 2025 Datacenter, 10 vCPUs, 64 GB RAM
  • VM 2 (PLOTRON 2.0): Windows Server 2025 Datacenter, 10 vCPUs, 64 GB RAM
  • Host workload (StandBy): Running natively on host OS, not a VM

VM Configuration

  • CPU Reserve: 20% per VM
  • AutoStart: Enabled with staggered delays (30s and 60s)
  • AutoStop: Shutdown
  • Enhanced Session: HvSocket
  • Dynamic Memory: Disabled (fixed allocation)

Software Stack per Instance (all three, host and both VMs)

  • Apache Solr 9.8.1 (SolrCloud with ZooKeeper ensemble)
  • MySQL 5.7
  • Custom .NET screening service (Tasker)
  • IIS for web portal

Network

  • Host: 175.111.0.252/29 (static public)
  • VM 1: 175.111.0.253 (public), 10.0.0.253 (LAN)
  • VM 2: 175.111.0.254 (public), 10.0.0.250 (LAN)
  • Gateway: 175.111.0.249 | DNS: 1.1.1.1, 8.8.8.8

Tools Used

  • CpuGroups.exe, Microsoft Download Center (Hyper-V CPU Groups utility)
  • PowerShell 5.1
  • Windows Performance Monitor (Get-Counter)

AI Assistance

This solution was worked out with the help of Claude Sonnet 4.5 by Anthropic across multiple diagnostic sessions. Claude helped narrow down the root cause, identify the VBS and scheduler obstacles, iterate on the PowerShell scripts, and draft this writeup. All commands and configurations were tested live on the hardware described above.

About the Author

Haris Ali Khan is Co-Founder and Business Head of AML/CFT Compliance Solutions at First Paramount Modaraba (FPM), a SECP-regulated, PSX-listed Islamic financial institution based in Karachi, Pakistan. He leads business development, client engagement, and product architecture for FPM AML-CHECK©, a proprietary compliance screening platform serving regulated financial institutions across Pakistan and international markets.

His work spans AML/CFT regulatory technology, server infrastructure management, Islamic finance, and Real World Asset (RWA) fractionalization technologies.

LinkedIn: https://pk.linkedin.com/in/haris-ali-khan

Top comments (0)