<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Aeza</title>
    <description>The latest articles on DEV Community by Aeza (@aeza__net).</description>
    <link>https://dev.to/aeza__net</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4108163%2F58f1152c-71c2-4496-8db7-c426fb9c26e7.jpg</url>
      <title>DEV Community: Aeza</title>
      <link>https://dev.to/aeza__net</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/aeza__net"/>
    <language>en</language>
    <item>
      <title>VPS Server: What It Is, How It Works, and Who Needs It</title>
      <dc:creator>Aeza</dc:creator>
      <pubDate>Mon, 14 Sep 2026 19:50:25 +0000</pubDate>
      <link>https://dev.to/aeza__net/vps-server-what-it-is-how-it-works-and-who-needs-it-534e</link>
      <guid>https://dev.to/aeza__net/vps-server-what-it-is-how-it-works-and-who-needs-it-534e</guid>
      <description>&lt;p&gt;A VPS server provides a separate operating system and administrative access without renting an entire physical machine.&lt;/p&gt;

&lt;p&gt;This solution is suitable for services requiring a permanent IP address and predictable workloads.&lt;/p&gt;

&lt;p&gt;When choosing a VPS, important factors include virtualization technology, CPU, memory, storage, network limits, and responsibility boundaries. Two plans with identical descriptions may have completely different real-world conditions.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is a VPS in simple terms?
&lt;/h2&gt;

&lt;p&gt;A VPS is a virtual machine with a guest operating system running on physical hardware through a hypervisor.&lt;/p&gt;

&lt;p&gt;The user receives:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;administrative access;&lt;/li&gt;
&lt;li&gt;an IP address;&lt;/li&gt;
&lt;li&gt;virtual disks;&lt;/li&gt;
&lt;li&gt;allocated resource limits.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The provider manages the physical host and virtualization layer.&lt;/p&gt;

&lt;p&gt;Actual isolation and performance guarantees depend on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;virtualization technology;&lt;/li&gt;
&lt;li&gt;resource allocation model;&lt;/li&gt;
&lt;li&gt;provider terms.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What is included in the service and what remains your responsibility?
&lt;/h2&gt;

&lt;p&gt;A physical server is divided into multiple virtual machines.&lt;/p&gt;

&lt;p&gt;Each VM runs its own operating system, filesystem, users, processes, and network configuration.&lt;/p&gt;

&lt;p&gt;Processes from one VM cannot access another VM's files or memory, although all machines share the same physical hardware.&lt;/p&gt;

&lt;p&gt;A VPS plan usually includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;vCPU;&lt;/li&gt;
&lt;li&gt;RAM;&lt;/li&gt;
&lt;li&gt;storage;&lt;/li&gt;
&lt;li&gt;IP address;&lt;/li&gt;
&lt;li&gt;traffic limit or network bandwidth.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The provider panel usually manages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;power operations;&lt;/li&gt;
&lt;li&gt;OS reinstallations;&lt;/li&gt;
&lt;li&gt;emergency console access.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Inside the operating system, administration remains your responsibility:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;web server;&lt;/li&gt;
&lt;li&gt;database;&lt;/li&gt;
&lt;li&gt;updates;&lt;/li&gt;
&lt;li&gt;users;&lt;/li&gt;
&lt;li&gt;backups.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Administrative access gives flexibility but also responsibility.&lt;/p&gt;

&lt;p&gt;A wrong SSH configuration, exposed database, or deleted file exists inside the guest system and will not be fixed by the hypervisor.&lt;/p&gt;

&lt;p&gt;Before ordering, verify:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;who installs updates;&lt;/li&gt;
&lt;li&gt;who handles incidents;&lt;/li&gt;
&lt;li&gt;who restores data.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How does a hypervisor create multiple servers?
&lt;/h2&gt;

&lt;p&gt;On Linux, KVM uses CPU virtualization extensions and provides an interface for running virtual machines.&lt;/p&gt;

&lt;p&gt;Each VM usually runs as a QEMU process.&lt;/p&gt;

&lt;p&gt;QEMU executes guest code through KVM and provides virtual hardware.&lt;/p&gt;

&lt;p&gt;The management layer, commonly libvirt, defines:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;vCPU;&lt;/li&gt;
&lt;li&gt;memory;&lt;/li&gt;
&lt;li&gt;disks;&lt;/li&gt;
&lt;li&gt;network interfaces;&lt;/li&gt;
&lt;li&gt;startup rules.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The virtualization stack:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Physical CPU, RAM, NVMe, NIC
        ↓
Linux kernel with KVM
        ↓
QEMU + virtio devices
        ↓
Guest OS
        ↓
Application
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The host scheduler distributes CPU time between virtual machines.&lt;/p&gt;

&lt;p&gt;CPU pinning, quotas, and overcommit depend on the provider configuration.&lt;/p&gt;

&lt;p&gt;The guest does not access hardware directly. It uses virtual resources provided by QEMU.&lt;/p&gt;

&lt;p&gt;Disk and network operations use paravirtualized virtio queues.&lt;/p&gt;

&lt;p&gt;Virtual machines are isolated from each other, but physical CPU, storage, and networking remain shared.&lt;/p&gt;

&lt;p&gt;Therefore, comparing VPS providers requires understanding resource allocation and testing real workloads.&lt;/p&gt;

&lt;h2&gt;
  
  
  What do vCPU, RAM, NVMe, IOPS, and bandwidth mean?
&lt;/h2&gt;

&lt;p&gt;vCPU shows how many virtual processors the guest sees.&lt;/p&gt;

&lt;p&gt;You should also evaluate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU generation;&lt;/li&gt;
&lt;li&gt;frequency;&lt;/li&gt;
&lt;li&gt;CPU quotas;&lt;/li&gt;
&lt;li&gt;host contention.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For continuous workloads, long-term performance and scheduling latency matter.&lt;/p&gt;

&lt;p&gt;Linux &lt;code&gt;%steal&lt;/code&gt; shows when a VM wanted CPU time but the hypervisor did not provide it.&lt;/p&gt;

&lt;p&gt;RAM defines available memory.&lt;/p&gt;

&lt;p&gt;Behavior during memory pressure depends on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;memory guarantees;&lt;/li&gt;
&lt;li&gt;ballooning;&lt;/li&gt;
&lt;li&gt;overcommit;&lt;/li&gt;
&lt;li&gt;swap policy.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Frequent swapping increases latency.&lt;/p&gt;

&lt;p&gt;For databases, guaranteed RAM with additional capacity is often better than a large advertised amount without guarantees.&lt;/p&gt;

&lt;p&gt;NVMe performance depends on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;IOPS;&lt;/li&gt;
&lt;li&gt;throughput;&lt;/li&gt;
&lt;li&gt;block size;&lt;/li&gt;
&lt;li&gt;queue depth;&lt;/li&gt;
&lt;li&gt;read/write ratio.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;3000 IOPS with 4 KiB blocks
≈ 11.7 MiB/s random workload
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Real performance is affected by filesystem overhead, queues, and contention.&lt;/p&gt;

&lt;p&gt;Network resources include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;port speed;&lt;/li&gt;
&lt;li&gt;traffic limits;&lt;/li&gt;
&lt;li&gt;packets per second;&lt;/li&gt;
&lt;li&gt;outgoing bandwidth;&lt;/li&gt;
&lt;li&gt;routing quality.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A cloud VPS may provide APIs and automatic resource creation, but scaling capacity depends on the actual infrastructure, not the product name.&lt;/p&gt;

&lt;h2&gt;
  
  
  VPS vs shared hosting vs dedicated server
&lt;/h2&gt;

&lt;p&gt;A VPS provides:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;its own operating system;&lt;/li&gt;
&lt;li&gt;administrative access;&lt;/li&gt;
&lt;li&gt;resource limits.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Shared hosting provides a managed environment with less control.&lt;/p&gt;

&lt;p&gt;A dedicated server provides the entire physical machine to one customer.&lt;/p&gt;

&lt;p&gt;Managed services, CPU guarantees, memory guarantees, and storage guarantees differ between providers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Managed vs unmanaged VPS
&lt;/h2&gt;

&lt;p&gt;With unmanaged VPS:&lt;/p&gt;

&lt;p&gt;The provider manages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;physical hardware;&lt;/li&gt;
&lt;li&gt;hypervisor;&lt;/li&gt;
&lt;li&gt;power;&lt;/li&gt;
&lt;li&gt;external network.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The customer manages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;OS;&lt;/li&gt;
&lt;li&gt;accounts;&lt;/li&gt;
&lt;li&gt;firewall;&lt;/li&gt;
&lt;li&gt;applications;&lt;/li&gt;
&lt;li&gt;updates;&lt;/li&gt;
&lt;li&gt;logs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The exact responsibility boundary depends on the contract.&lt;/p&gt;

&lt;p&gt;Managed VPS may include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;monitoring;&lt;/li&gt;
&lt;li&gt;updates;&lt;/li&gt;
&lt;li&gt;alerts;&lt;/li&gt;
&lt;li&gt;recovery assistance.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;However, the exact service must be checked before purchase.&lt;/p&gt;

&lt;p&gt;A control panel simplifies management but does not automatically make a VPS managed.&lt;/p&gt;

&lt;p&gt;Snapshots are not always backups because they may exist on the same platform and may not guarantee application recovery.&lt;/p&gt;

&lt;h2&gt;
  
  
  Common VPS use cases
&lt;/h2&gt;

&lt;p&gt;VPS servers are commonly used for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;websites;&lt;/li&gt;
&lt;li&gt;APIs;&lt;/li&gt;
&lt;li&gt;bots;&lt;/li&gt;
&lt;li&gt;monitoring systems;&lt;/li&gt;
&lt;li&gt;Git runners;&lt;/li&gt;
&lt;li&gt;testing environments.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;They are useful when applications require:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a permanent IP;&lt;/li&gt;
&lt;li&gt;background processes;&lt;/li&gt;
&lt;li&gt;operating system control.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Heavy databases and workloads requiring GPUs may require different solutions.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Workload&lt;/th&gt;
&lt;th&gt;Example profile&lt;/th&gt;
&lt;th&gt;Critical metric&lt;/th&gt;
&lt;th&gt;Suitable option&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Website/API&lt;/td&gt;
&lt;td&gt;2 vCPU, 4 GB RAM&lt;/td&gt;
&lt;td&gt;p95 latency, RAM&lt;/td&gt;
&lt;td&gt;VPS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Database&lt;/td&gt;
&lt;td&gt;High I/O workload&lt;/td&gt;
&lt;td&gt;IOPS, disk latency, RAM guarantees&lt;/td&gt;
&lt;td&gt;VPS with guarantees or dedicated&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bots&lt;/td&gt;
&lt;td&gt;Small CPU, constant network&lt;/td&gt;
&lt;td&gt;bandwidth, PPS&lt;/td&gt;
&lt;td&gt;VPS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPU workloads&lt;/td&gt;
&lt;td&gt;CUDA/ROCm&lt;/td&gt;
&lt;td&gt;GPU model, VRAM&lt;/td&gt;
&lt;td&gt;GPU instance/dedicated&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Traffic spikes&lt;/td&gt;
&lt;td&gt;Rapid RPS growth&lt;/td&gt;
&lt;td&gt;scaling speed&lt;/td&gt;
&lt;td&gt;Cloud infrastructure&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Testing is still required because the same hardware can behave differently depending on workload.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to choose a VPS plan
&lt;/h2&gt;

&lt;p&gt;Start with measurements.&lt;/p&gt;

&lt;p&gt;For CPU:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;usage per core;&lt;/li&gt;
&lt;li&gt;saturation time;&lt;/li&gt;
&lt;li&gt;request latency.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For memory:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;working set;&lt;/li&gt;
&lt;li&gt;page faults;&lt;/li&gt;
&lt;li&gt;swap usage.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For storage:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;IOPS;&lt;/li&gt;
&lt;li&gt;throughput;&lt;/li&gt;
&lt;li&gt;p95/p99 latency.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For network:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;average speed;&lt;/li&gt;
&lt;li&gt;peak speed;&lt;/li&gt;
&lt;li&gt;PPS;&lt;/li&gt;
&lt;li&gt;outgoing traffic.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If no baseline exists, start with the smallest suitable plan and run repeatable tests.&lt;/p&gt;

&lt;p&gt;Upgrade only the resource that becomes the bottleneck.&lt;/p&gt;

&lt;p&gt;A plan such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2 vCPU
4 GB RAM
60 GB storage
3000 IOPS
100 Mbps port
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;is not enough information without knowing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU model;&lt;/li&gt;
&lt;li&gt;CPU quota;&lt;/li&gt;
&lt;li&gt;storage type;&lt;/li&gt;
&lt;li&gt;throughput limits;&lt;/li&gt;
&lt;li&gt;traffic rules.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The name VPS or VDS alone does not describe virtualization quality.&lt;/p&gt;

&lt;h2&gt;
  
  
  Suitable VPS tasks
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Websites, APIs, and small databases with stable workloads.&lt;/li&gt;
&lt;li&gt;Bots, monitoring systems, Git runners, and test environments.&lt;/li&gt;
&lt;li&gt;Applications requiring their own OS and administrative access.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Initial VPS security setup
&lt;/h2&gt;

&lt;p&gt;After first connection:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Verify the SSH fingerprint through a trusted channel.&lt;/li&gt;
&lt;li&gt;Create a separate administrator account.&lt;/li&gt;
&lt;li&gt;Add a public SSH key.&lt;/li&gt;
&lt;li&gt;Test key-based login.&lt;/li&gt;
&lt;li&gt;Disable password authentication and root login only after recovery access is confirmed.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Configure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;firewall rules;&lt;/li&gt;
&lt;li&gt;OS updates;&lt;/li&gt;
&lt;li&gt;application updates;&lt;/li&gt;
&lt;li&gt;least-privilege services.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A VPS isolates the machine, not the software.&lt;/p&gt;

&lt;p&gt;A vulnerable plugin, leaked token, or incorrect access rule remains your responsibility.&lt;/p&gt;

&lt;p&gt;Backups should survive:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;VM loss;&lt;/li&gt;
&lt;li&gt;account loss;&lt;/li&gt;
&lt;li&gt;infrastructure failure.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Store backups separately, encrypt them, and regularly test restoration.&lt;/p&gt;

&lt;p&gt;Monitoring should alert about:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;disk usage;&lt;/li&gt;
&lt;li&gt;stopped services;&lt;/li&gt;
&lt;li&gt;errors;&lt;/li&gt;
&lt;li&gt;expiring certificates.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Critical systems need a recovery procedure describing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;console access;&lt;/li&gt;
&lt;li&gt;backup location;&lt;/li&gt;
&lt;li&gt;rollback process;&lt;/li&gt;
&lt;li&gt;recovery validation.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  VPS, VDS, cloud, and dedicated servers
&lt;/h2&gt;

&lt;p&gt;VPS and VDS are often marketing terms.&lt;/p&gt;

&lt;p&gt;The name alone does not show:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU allocation;&lt;/li&gt;
&lt;li&gt;overcommit policy;&lt;/li&gt;
&lt;li&gt;storage guarantees;&lt;/li&gt;
&lt;li&gt;virtualization method.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Evaluate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU model;&lt;/li&gt;
&lt;li&gt;quotas;&lt;/li&gt;
&lt;li&gt;memory guarantees;&lt;/li&gt;
&lt;li&gt;IOPS;&lt;/li&gt;
&lt;li&gt;network limits;&lt;/li&gt;
&lt;li&gt;SLA.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cloud platforms are useful when infrastructure must be created through APIs, distributed across zones, or connected with managed services.&lt;/p&gt;

&lt;p&gt;Dedicated servers provide the entire physical machine and are suitable for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;high constant workloads;&lt;/li&gt;
&lt;li&gt;large databases;&lt;/li&gt;
&lt;li&gt;specialized hardware requirements.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The correct choice depends on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;required control;&lt;/li&gt;
&lt;li&gt;recovery speed;&lt;/li&gt;
&lt;li&gt;workload stability;&lt;/li&gt;
&lt;li&gt;downtime cost.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A VPS server is suitable when a project needs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;its own operating system;&lt;/li&gt;
&lt;li&gt;network control;&lt;/li&gt;
&lt;li&gt;predictable resources;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;but a dedicated physical machine is unnecessary.&lt;/p&gt;

&lt;p&gt;Before purchasing, verify technical parameters, responsibility boundaries, and recovery procedures.&lt;/p&gt;

&lt;p&gt;If monitoring confirms that the workload fits within limits with sufficient resource reserve, a VPS can provide the required infrastructure without unnecessary complexity.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>programming</category>
      <category>python</category>
      <category>devops</category>
    </item>
    <item>
      <title>Choosing a Game Server Location Based on Ping</title>
      <dc:creator>Aeza</dc:creator>
      <pubDate>Mon, 14 Sep 2026 19:44:28 +0000</pubDate>
      <link>https://dev.to/aeza__net/choosing-a-game-server-location-based-on-ping-3dhi</link>
      <guid>https://dev.to/aeza__net/choosing-a-game-server-location-based-on-ping-3dhi</guid>
      <description>&lt;p&gt;Choosing a game server location based on ping is determined not by distance alone, but by the networks used by future players. A nearby city may perform worse if packets reach it through an overloaded interconnection between operators. A proper comparison requires several networks, different times of day, and a test gameplay session, because service-level ping requests may behave differently from actual game traffic. Measurements should not be taken only from your own network. Instead, players themselves should run them: several people from each city and from each ISP run the same test, and you aggregate the results.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which latency tests should be done before choosing a location?
&lt;/h3&gt;

&lt;p&gt;Before choosing a location, you need ping series and traceroutes from player networks, followed by a test session on the game server. Together, these help evaluate latency, jitter, and packet loss at different times of day. Responses to service-level requests may differ from the behavior of actual game traffic, so ping alone is not enough.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which players are we choosing the server for?
&lt;/h2&gt;

&lt;p&gt;Player groups differ by country, city, ISP, and connection type. Even neighbors using different providers may have different routes to the server. Therefore, measurements must be taken from the actual networks used by your audience. ASN, the Autonomous System Number, helps distinguish networks.&lt;/p&gt;

&lt;p&gt;Testing game server location latency should rely on several volunteers from each group. Their results can be stored under codes such as &lt;code&gt;G1-P1&lt;/code&gt; and &lt;code&gt;G1-P2&lt;/code&gt;, along with the city, ISP, and connection method. If a player uses Wi-Fi in practice, they should measure the same connection they actually use. If one player’s result is significantly worse than the others in the same group, an additional wired test can show whether the issue is in the home network or in the route itself.&lt;/p&gt;

&lt;p&gt;Before measuring, define thresholds: what latency, what packet loss rate, and how many session disconnects are considered acceptable. These thresholds depend on the game and the expectations of the community. Requirements should account for every significant group, because a good average can easily hide problems affecting a minority.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which servers should be compared?
&lt;/h2&gt;

&lt;p&gt;Location should remain the only difference between candidates. The test VMs should match in CPU resources, network limits, game version, and firewall settings. On an overloaded machine, latency may be caused by resource limitations rather than the network. The candidate list should include the domain name, actual IP, game port, and the period of access to the test environment.&lt;/p&gt;

&lt;p&gt;Providers often publish a test IP that can be pinged before renting a location. That is enough to rule out obviously distant regions, but not enough to choose between nearby ones. It is worth asking support whether the address belongs to the required site and whether it passes through the same DDoS protection path as the future VM. Final measurements should be taken on the rented machine.&lt;/p&gt;

&lt;p&gt;Ping, jitter, and packet loss answer different questions: how quickly a response arrives, how stable its timing is, and how many responses never arrive at all. A low average can coexist with spikes and losses. IPv4 and IPv6 routes may differ, so if both protocols are supported, they should be tested separately.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to collect comparable measurements
&lt;/h2&gt;

&lt;p&gt;A comparison will show which country to choose for a VPS only if every participant performs the same scenario for all candidates. For an initial selection, several five-minute series are sufficient: in the morning, in the evening, and during usual gaming hours. If results are close, additional days of observation are required.&lt;/p&gt;

&lt;p&gt;The script below is intended for Linux with Bash, the &lt;code&gt;ping&lt;/code&gt; utility from the &lt;code&gt;iputils&lt;/code&gt; package, and MTR. Save the code as &lt;code&gt;test-region.sh&lt;/code&gt;. In the launch command, the IP should be the address of the test VM. Results and utility versions will be saved into a separate folder with a UTC timestamp.&lt;/p&gt;

&lt;p&gt;If MTR refuses to open sockets, run the script with &lt;code&gt;sudo&lt;/code&gt;, because some builds require root privileges.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# Usage: bash test-region.sh IP&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C
&lt;span class="nv"&gt;ip&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;:?Specify&lt;span class="p"&gt; the IPv4 address of the test VM&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;mktemp&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s2"&gt;"probe-&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; +%Y%m%dT%H%M%SZ&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;-XXXX"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'%s\n'&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ip&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;/target.txt"&lt;/span&gt;
ping &lt;span class="nt"&gt;-V&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;/versions.txt"&lt;/span&gt; 2&amp;gt;&amp;amp;1
mtr &lt;span class="nt"&gt;--version&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;/versions.txt"&lt;/span&gt; 2&amp;gt;&amp;amp;1
ping &lt;span class="nt"&gt;-4&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="nt"&gt;-D&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; 300 &lt;span class="nt"&gt;-i&lt;/span&gt; 1 &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ip&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;/ping.txt"&lt;/span&gt; 2&amp;gt;&amp;amp;1
mtr &lt;span class="nt"&gt;-4&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="nt"&gt;-w&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; 60 &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ip&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;/mtr.txt"&lt;/span&gt; 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ping will send 300 requests at one-second intervals, then MTR will collect a route report. The whole run will take more than five minutes. The order of candidates should be changed between runs so that one region is not always tested before the others. Command parameters are described in the &lt;a href="https://man7.org/linux/man-pages/man8/ping.8.html" rel="noopener noreferrer"&gt;ping documentation&lt;/a&gt; and the &lt;a href="https://github.com/traviscross/mtr/blob/master/man/mtr.8.in" rel="noopener noreferrer"&gt;MTR manual&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  How much more important are jitter and packet loss than average ping?
&lt;/h3&gt;

&lt;p&gt;A low average ping does not compensate for frequent latency spikes and packet loss. Because of them, data arrives unevenly or does not reach the game at all, even if most responses are fast. The importance of each metric depends on the game mechanics and how it handles loss, so the final comparison requires testing a real gameplay session.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the numbers in the report mean
&lt;/h2&gt;

&lt;p&gt;RTT is the time from sending a request to receiving a response. The median, or p50, divides the measured values into two halves. The p95 value indicates the latency not exceeded by about 95% of the received responses. This makes slow responses visible even when the average looks good. The number of requests and responses is needed to evaluate the sample.&lt;/p&gt;

&lt;p&gt;For consistency, p95 can be calculated as follows: sort RTT values in ascending order and take the value at position &lt;code&gt;0.95 × N&lt;/code&gt;, rounded up, where &lt;code&gt;N&lt;/code&gt; is the number of responses. A short series is not enough to draw confident conclusions about p99. A graph of RTT over time will show whether spikes were concentrated in a single episode.&lt;/p&gt;

&lt;p&gt;The term &lt;em&gt;jitter&lt;/em&gt; is used for different measures of latency variation. Here it means the average absolute RTT difference between consecutive requests for which both have a response. This calculation is neither the same as &lt;code&gt;mdev&lt;/code&gt; from ping output nor the one-way delay variation defined in &lt;a href="https://www.rfc-editor.org/rfc/rfc3393.html" rel="noopener noreferrer"&gt;RFC 3393&lt;/a&gt;. Numbers from different sources should not be compared directly.&lt;/p&gt;

&lt;p&gt;The percentage of unanswered requests is &lt;code&gt;(tx − rx) / tx × 100%&lt;/code&gt;, where &lt;code&gt;tx&lt;/code&gt; and &lt;code&gt;rx&lt;/code&gt; are the number of requests and unique responses. A lost request is neither turned into zero RTT nor into RTT equal to the timeout. A hosting provider’s looking glass can be used to check connectivity from the provider side, but player-side measurements are still necessary.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to investigate an unexpected route
&lt;/h2&gt;

&lt;p&gt;If one region performs significantly worse, a traceroute to the game server will help reveal the responding intermediate hops. MTR repeats such probes and collects statistics. Reports from several networks during the problematic period will show whether the degradation affects only one operator or several.&lt;/p&gt;

&lt;p&gt;Asterisks and high loss percentages on an intermediate hop do not automatically mean that it is dropping forwarded traffic. A router may rate-limit service replies while still forwarding packets normally. The suspicion becomes stronger when degradation is also visible at the final destination. The actual cause of the issue should be confirmed by the operator.&lt;/p&gt;

&lt;p&gt;Hop names may hint at transit through another city, although geography inferred from names or IP databases may be inaccurate. A reverse traceroute from the server to a reachable player address provides information about the opposite direction. Forward and reverse paths may differ, and RTT includes both. A home address hidden behind NAT may limit such verification.&lt;/p&gt;

&lt;h2&gt;
  
  
  Testing inside the game itself
&lt;/h2&gt;

&lt;p&gt;After network probes, a gameplay session is required on every candidate. The servers should match in game version, mods, map, and settings. Participants should ideally repeat similar actions with the same number of players. The report should include client-side latency, disconnects, and any available packet loss indicators. Some games calculate built-in ping using their own smoothing and internal processing, so it does not have to match ICMP RTT.&lt;/p&gt;

&lt;p&gt;VM load requires separate observation. For example, in Minecraft, MSPT means milliseconds per tick, while FPS refers to the client’s frame rate. Slow ticks or FPS drops can ruin the experience even if the network is fine, so these metrics should be stored separately from RTT.&lt;/p&gt;

&lt;p&gt;DDoS protection may change the traffic path. Some providers reroute traffic through a &lt;a href="https://blog.cloudflare.com/no-scrubs-architecture-unmetered-mitigation/" rel="noopener noreferrer"&gt;scrubbing center&lt;/a&gt; — a dedicated site where packets are filtered before being sent to the server. Others filter directly on edge nodes, in which case no visible detour appears. Any such testing must be coordinated with the provider, without independently organizing attacks. After a known enablement or change in filtering, it is useful to repeat both the gameplay session and the network probes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why can the closest region on the map be slower?
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Traffic exchange between operators, or peering, may go through a distant point.&lt;/li&gt;
&lt;li&gt;A congested segment on the route adds queueing delay even when the physical distance is small.&lt;/li&gt;
&lt;li&gt;Traffic filtering may change the route to the server and increase latency.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  How to choose based on the results
&lt;/h2&gt;

&lt;p&gt;In the table, each combination of player group, candidate region, and test time should have its own row: a good daytime run must not hide a bad evening one. The four main columns are p50, p95, RTT variation, and packet loss.&lt;/p&gt;

&lt;p&gt;In the summary CSV, the &lt;code&gt;group&lt;/code&gt; field links the row to a city and operator, &lt;code&gt;trace&lt;/code&gt; stores the traceroute filename, and &lt;code&gt;game&lt;/code&gt; contains the result of the gameplay session.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;player,group,region,utc,tx,rx,p50,p95,jitter,loss,trace,game
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;player&lt;/code&gt; field stores a participant code. Latencies are in milliseconds, and &lt;code&gt;loss&lt;/code&gt; is in percent. One row corresponds to one measurement series. An overall p95 cannot be obtained by averaging the p95 values of individual participants.&lt;/p&gt;

&lt;p&gt;Candidates that violate mandatory requirements for any significant group are removed from the comparison. The remaining candidates are scored using predefined scales for each metric, for example from 0 to 5, where 5 means the best result. Weights should be defined before calculation. An example is shown below and should be adapted to your own game.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;strong&gt;Metric&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Weight&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;What is evaluated&lt;/strong&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;p50 RTT&lt;/td&gt;
&lt;td&gt;20%&lt;/td&gt;
&lt;td&gt;Typical latency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p95 RTT&lt;/td&gt;
&lt;td&gt;40%&lt;/td&gt;
&lt;td&gt;Slow responses&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RTT variation&lt;/td&gt;
&lt;td&gt;20%&lt;/td&gt;
&lt;td&gt;Stability of latency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Packet loss&lt;/td&gt;
&lt;td&gt;20%&lt;/td&gt;
&lt;td&gt;Unanswered requests&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The score of each metric is multiplied by its weight, and the sum gives the group score. Group scores are then weighted according to each group’s share of the audience. If scores are close, route stability becomes the deciding factor: a routing path that changes transit provider from one series to another is riskier than a stable path, even with the same p95. The availability of measurement endpoints also matters, because a region that cannot be rechecked after launch is less reliable in practice. If no single region is suitable for everyone, it may be worth considering several servers for different groups.&lt;/p&gt;

&lt;h2&gt;
  
  
  When to repeat the tests
&lt;/h2&gt;

&lt;p&gt;After launch, the same participants and the same scenario help verify quality on the permanent IP. New measurements are required after changes to the address, provider, filtering settings, or when complaints appear from a specific network.&lt;/p&gt;

&lt;p&gt;Player geography also changes. A new large group may change the optimal choice even if route quality remains the same. Therefore, the list of cities, ISPs, and the shares of active participants should be updated regularly.&lt;/p&gt;

&lt;p&gt;The report should include the author, dates and time zone, test addresses, VM configuration, utility versions, and game version. A summary CSV without raw data cannot be rechecked, so RTT series, traceroutes, and gameplay notes should be preserved alongside it. Before publication, home IPs and other personal data should be removed from the logs. The measurement date should be shown next to the results. Routing changes over time, so output becomes outdated as the network evolves.&lt;/p&gt;

&lt;p&gt;Choosing a game server location based on ping should end with a decision that clearly states the conditions: which groups the region is suitable for, where limitations remain, and when the verification took place. If results are close, the test should be extended. Before the final migration, it is wise to preserve the ability to return to the previous location in case the new route proves unstable.&lt;/p&gt;

</description>
      <category>gamedev</category>
      <category>networking</category>
      <category>performance</category>
    </item>
    <item>
      <title>SSH Hardening Checklist: How to Configure Access and Verify the Result</title>
      <dc:creator>Aeza</dc:creator>
      <pubDate>Thu, 10 Sep 2026 12:23:51 +0000</pubDate>
      <link>https://dev.to/aeza__net/ssh-hardening-checklist-how-to-configure-access-and-verify-the-result-19ee</link>
      <guid>https://dev.to/aeza__net/ssh-hardening-checklist-how-to-configure-access-and-verify-the-result-19ee</guid>
      <description>&lt;p&gt;An SSH hardening checklist helps eliminate unnecessary login methods on a VPS. Hardening means configuring the server to reduce unnecessary exposure and improve access security. A secure &lt;code&gt;sshd_config&lt;/code&gt; setup comes down to leaving only the authentication methods that are actually required and making sure access can be restored if something goes wrong.&lt;/p&gt;

&lt;p&gt;The examples below are intended for Ubuntu 24.04 LTS with OpenSSH from the operating system repositories. If you use another OS or have specific audit requirements, verify the relevant parameters separately.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which sshd_config parameters matter most during an audit?
&lt;/h3&gt;

&lt;p&gt;The main parameters are &lt;code&gt;PermitRootLogin&lt;/code&gt; for direct root access, &lt;code&gt;PasswordAuthentication&lt;/code&gt; and &lt;code&gt;KbdInteractiveAuthentication&lt;/code&gt; for authentication methods, and &lt;code&gt;AllowUsers&lt;/code&gt; or &lt;code&gt;AllowGroups&lt;/code&gt; for restricting who is allowed to connect.&lt;/p&gt;

&lt;p&gt;When multi-factor authentication is used, &lt;code&gt;AuthenticationMethods&lt;/code&gt; becomes important because it defines which authentication methods must succeed before access is granted.&lt;/p&gt;

&lt;p&gt;The configuration should be verified with real login attempts for both permitted and prohibited scenarios.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Exactly Needs to Be Protected
&lt;/h2&gt;

&lt;p&gt;A secure &lt;code&gt;sshd_config&lt;/code&gt; configuration starts with identifying everyone who actually needs SSH access: the VPS owner, colleagues, automation, backup systems, and other service accounts.&lt;/p&gt;

&lt;p&gt;Passwords can be brute-forced, keys can be stolen, and former employees may retain working access long after it should have been revoked.&lt;/p&gt;

&lt;p&gt;Key-only authentication removes password guessing as an attack path. A stolen key still has to be revoked, unnecessary accounts need to be disabled, and a second factor such as a one-time code can reduce the risk associated with a compromised key.&lt;/p&gt;

&lt;p&gt;Another risk is losing trustworthy logs. If SSH authentication events exist only on the same server, an attacker who obtains root privileges may modify or remove them, making later incident reconstruction much harder.&lt;/p&gt;

&lt;p&gt;Configuration errors can also lock out the administrator who is applying the hardening.&lt;/p&gt;

&lt;p&gt;Before changing anything, verify that the provider console gives you administrative access to the filesystem or an equivalent recovery mechanism. In a team environment, it may be useful to explicitly assign someone responsibility for confirming that recovery access exists.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check Which Configuration the Server Is Actually Using
&lt;/h2&gt;

&lt;p&gt;An SSH audit checklist should include the operating system version, OpenSSH version, and effective configuration before any changes are made.&lt;/p&gt;

&lt;p&gt;You can inspect them with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /etc/os-release
&lt;span class="nb"&gt;sudo&lt;/span&gt; /usr/sbin/sshd &lt;span class="nt"&gt;-V&lt;/span&gt;
&lt;span class="nb"&gt;sudo&lt;/span&gt; /usr/sbin/sshd &lt;span class="nt"&gt;-T&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Ubuntu, &lt;code&gt;/etc/ssh/sshd_config&lt;/code&gt; includes files from &lt;code&gt;/etc/ssh/sshd_config.d/&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Usually, the first value read for a parameter takes precedence, and the order depends on the names of included files. Because of that, adding a setting near the end of the main configuration file may not produce the result you expect.&lt;/p&gt;

&lt;p&gt;More details are available in the &lt;a href="https://ubuntu.com/server/docs/how-to/security/openssh-server/" rel="noopener noreferrer"&gt;Ubuntu OpenSSH documentation&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Match&lt;/code&gt; blocks create exceptions for specific users, addresses, or other connection properties.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;-C&lt;/code&gt; option lets you evaluate the effective configuration for a particular connection:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo&lt;/span&gt; /usr/sbin/sshd &lt;span class="nt"&gt;-T&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-C&lt;/span&gt; &lt;span class="nv"&gt;user&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;admin,addr&lt;span class="o"&gt;=&lt;/span&gt;198.51.100.10,host&lt;span class="o"&gt;=&lt;/span&gt;client.example

&lt;span class="nb"&gt;sudo &lt;/span&gt;ss &lt;span class="nt"&gt;-ltnp&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Replace the username, client address, and hostname with the values from your environment.&lt;/p&gt;

&lt;p&gt;If a &lt;code&gt;Match&lt;/code&gt; condition also depends on the server address or port, include &lt;code&gt;laddr&lt;/code&gt; and &lt;code&gt;lport&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The output of &lt;code&gt;sshd -T&lt;/code&gt; represents the configuration read from files on disk.&lt;/p&gt;

&lt;p&gt;Existing SSH sessions continue operating under the conditions established when they were created, so changes must be tested using a new connection.&lt;/p&gt;

&lt;p&gt;Listening ports shown by &lt;code&gt;ss&lt;/code&gt; should also be compared against firewall rules on the VPS and any network filtering configured in the provider control panel.&lt;/p&gt;

&lt;h2&gt;
  
  
  Identify Who Owns Every Account and SSH Key
&lt;/h2&gt;

&lt;p&gt;Create an inventory of all accounts that can log in and execute commands.&lt;/p&gt;

&lt;p&gt;Separately identify users that can obtain administrative privileges through &lt;code&gt;sudo&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;authorized_keys&lt;/code&gt; file contains public keys that are permitted to authenticate. Every key should have a known owner. A comment written next to a public key is useful metadata, but it does not prove who actually controls the corresponding private key.&lt;/p&gt;

&lt;p&gt;Disabling SSH password login should generally happen only after the account and key inventory has been completed.&lt;/p&gt;

&lt;p&gt;Personal administrator accounts make it easier to distinguish individual users in logs.&lt;/p&gt;

&lt;p&gt;Automation should use dedicated keys. For example, a backup process can have its own key rather than sharing an administrator credential.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;command=&lt;/code&gt; option in &lt;code&gt;authorized_keys&lt;/code&gt; can restrict a key to a specific command, while &lt;code&gt;restrict&lt;/code&gt; disables additional capabilities, including forwarding.&lt;/p&gt;

&lt;p&gt;After configuring these restrictions, verify that the intended automation still works.&lt;/p&gt;

&lt;p&gt;When rotating an SSH key, first install and test the replacement key. Only after confirming that the new key works should the previous one be removed from all servers.&lt;/p&gt;

&lt;p&gt;Deleting a key does not terminate sessions that were authenticated earlier, so compromised active sessions may require separate investigation and termination.&lt;/p&gt;

&lt;h3&gt;
  
  
  How Do You Disable Password Login Without Losing Access?
&lt;/h3&gt;

&lt;p&gt;First open a new SSH session using a key with a dedicated user that has working &lt;code&gt;sudo&lt;/code&gt; access.&lt;/p&gt;

&lt;p&gt;Keep the existing session open.&lt;/p&gt;

&lt;p&gt;Before applying any changes, validate the configuration, verify provider console access, and prepare a rollback procedure.&lt;/p&gt;

&lt;p&gt;After applying the new settings, open another new connection and confirm that key authentication still works.&lt;/p&gt;

&lt;p&gt;If multi-factor authentication is enabled, verify the second factor as well.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decide Which Authentication Methods Should Remain
&lt;/h2&gt;

&lt;p&gt;The following example permits authentication only with public keys.&lt;/p&gt;

&lt;p&gt;Replace &lt;code&gt;admin&lt;/code&gt; with the username for which you have already tested SSH access and &lt;code&gt;sudo&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Key-only authentication; no second factor through PAM
PubkeyAuthentication yes
AuthenticationMethods publickey
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
AllowUsers admin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;AllowUsers&lt;/code&gt; should contain every human administrator and service account that genuinely needs access.&lt;/p&gt;

&lt;p&gt;If group-based management is more convenient, use &lt;code&gt;AllowGroups&lt;/code&gt; instead.&lt;/p&gt;

&lt;p&gt;MFA and an SSH allowlist can work together: the allowlist restricts which accounts are eligible to connect, while the second factor reduces the risk of a stolen key.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;PermitRootLogin no&lt;/code&gt; disables direct root login completely.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;PermitRootLogin prohibit-password&lt;/code&gt;, by contrast, still permits root authentication with a public key.&lt;/p&gt;

&lt;p&gt;If authentication requires both a public key and a one-time code through PAM, use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;UsePAM yes
KbdInteractiveAuthentication yes
AuthenticationMethods publickey,keyboard-interactive:pam
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This form of multi-factor authentication only works if an appropriate PAM module has already been configured.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;keyboard-interactive&lt;/code&gt; itself does not necessarily mean MFA. Depending on the PAM stack, it may still prompt for a normal password.&lt;/p&gt;

&lt;p&gt;For that reason, setting only:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PasswordAuthentication no
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;is not sufficient to guarantee that every password-based authentication path has been disabled.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Apply Changes and Roll Them Back Safely
&lt;/h2&gt;

&lt;p&gt;Before modifying the server, verify key-based login from a new connection.&lt;/p&gt;

&lt;p&gt;Run the following from your own machine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nv"&gt;ControlPath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;none &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nv"&gt;PreferredAuthentications&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;publickey &lt;span class="se"&gt;\&lt;/span&gt;
  admin@SERVER
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This creates a new connection and permits only public-key authentication.&lt;/p&gt;

&lt;p&gt;If MFA through PAM is used, specify:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;publickey,keyboard-interactive
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;in &lt;code&gt;PreferredAuthentications&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Inside the new session, verify that &lt;code&gt;sudo&lt;/code&gt; works.&lt;/p&gt;

&lt;p&gt;Keep the old session open.&lt;/p&gt;

&lt;p&gt;In this example, configuration changes are placed in a new file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/etc/ssh/sshd_config.d/00-local-access.conf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Make sure both this name and the corresponding &lt;code&gt;.disabled&lt;/code&gt; filename are unused.&lt;/p&gt;

&lt;p&gt;Save the output of &lt;code&gt;sshd -T&lt;/code&gt; before making changes, then compare the effective values after modification, including any relevant &lt;code&gt;Match&lt;/code&gt; conditions.&lt;/p&gt;

&lt;p&gt;You should also determine how SSH is being started on the system.&lt;/p&gt;

&lt;p&gt;Run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl is-enabled ssh.socket
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Ubuntu, socket activation has been enabled by default since version 22.10.&lt;/p&gt;

&lt;p&gt;When socket activation is used, &lt;code&gt;sshd&lt;/code&gt; is started for incoming connections and reads the configuration for new connections, so a service reload is not required. Testing with a new session is enough.&lt;/p&gt;

&lt;p&gt;If socket activation is disabled and a persistent service is running, apply the changes with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl is-enabled ssh.socket
&lt;span class="nb"&gt;sudo&lt;/span&gt; /usr/sbin/sshd &lt;span class="nt"&gt;-t&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl reload ssh.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;sshd -t&lt;/code&gt; prints nothing when the configuration is valid.&lt;/p&gt;

&lt;p&gt;If validation fails, the command after &lt;code&gt;&amp;amp;&amp;amp;&lt;/code&gt; will not execute.&lt;/p&gt;

&lt;p&gt;Afterward, verify the root login restriction, normal key authentication, &lt;code&gt;sudo&lt;/code&gt;, and every login scenario that should be denied.&lt;/p&gt;

&lt;p&gt;For rollback, rename the added configuration file from the existing session or provider console so that it no longer matches &lt;code&gt;*.conf&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo mv&lt;/span&gt; /etc/ssh/sshd_config.d/00-local-access.conf &lt;span class="se"&gt;\&lt;/span&gt;
  /etc/ssh/sshd_config.d/00-local-access.conf.disabled

&lt;span class="nb"&gt;sudo&lt;/span&gt; /usr/sbin/sshd &lt;span class="nt"&gt;-t&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This rollback affects only the configuration file introduced in this example.&lt;/p&gt;

&lt;p&gt;PAM settings, firewall rules, and SSH port changes need their own rollback procedures.&lt;/p&gt;

&lt;p&gt;Do not close the original working SSH session until every expected login and rollback scenario has been tested successfully.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to Do With Tunneling and Cryptographic Algorithms
&lt;/h2&gt;

&lt;p&gt;SSH security on a VPS also depends on what users are allowed to do after authentication.&lt;/p&gt;

&lt;p&gt;An SSH tunnel forwards another application's traffic through the encrypted SSH connection.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;AllowTcpForwarding&lt;/code&gt; controls TCP forwarding.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;X11Forwarding&lt;/code&gt; allows graphical applications running on the server to display windows on the client.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;AllowAgentForwarding&lt;/code&gt; allows processes on the server to interact with the user's local SSH agent.&lt;/p&gt;

&lt;p&gt;Capabilities that are not required should be disabled.&lt;/p&gt;

&lt;p&gt;However, these settings should not be treated as complete containment. A user with a normal shell may be able to implement equivalent traffic forwarding using another program.&lt;/p&gt;

&lt;p&gt;With a supported OpenSSH version, it is usually better to start from the default cryptographic settings.&lt;/p&gt;

&lt;p&gt;Before manually changing cipher, MAC, or key exchange algorithm lists, verify client compatibility.&lt;/p&gt;

&lt;p&gt;Regular updates are generally more useful than maintaining a custom cryptographic configuration without a specific requirement.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;MaxAuthTries&lt;/code&gt; limits authentication attempts within one connection.&lt;/p&gt;

&lt;p&gt;Setting it too low may cause legitimate clients to fail before they have a chance to offer the correct key.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;ClientAliveInterval&lt;/code&gt; controls how often the server checks whether the client is still reachable. A user simply pausing while typing commands does not by itself cause the connection to be terminated.&lt;/p&gt;

&lt;h3&gt;
  
  
  Are Fail2ban and a Non-Standard SSH Port Enough?
&lt;/h3&gt;

&lt;p&gt;No.&lt;/p&gt;

&lt;p&gt;Moving SSH to another port often reduces automated login attempts, but a port scan can still discover the service.&lt;/p&gt;

&lt;p&gt;Blocking addresses after repeated authentication failures can reduce brute-force traffic, but it does not help when an attacker already has a valid private key.&lt;/p&gt;

&lt;p&gt;A stolen working key can authenticate successfully on the first attempt.&lt;/p&gt;

&lt;p&gt;Proper SSH hardening therefore requires more than hiding the port or installing Fail2ban.&lt;/p&gt;

&lt;p&gt;You still need account and key management, access restrictions, updates, reliable logging, and, depending on the risk model, multi-factor authentication.&lt;/p&gt;

&lt;h2&gt;
  
  
  Restrict the Network and Preserve Authentication Events
&lt;/h2&gt;

&lt;p&gt;A firewall can allow SSH only from administrator networks.&lt;/p&gt;

&lt;p&gt;Test access both from an allowed network and from an external network that should be rejected.&lt;/p&gt;

&lt;p&gt;Do not forget IPv6 if it is enabled.&lt;/p&gt;

&lt;p&gt;Fail2ban can temporarily block addresses after repeated failed authentication attempts.&lt;/p&gt;

&lt;p&gt;After installation, verify that it is actually reading SSH logs and applying bans.&lt;/p&gt;

&lt;p&gt;Be careful when testing: your own failed login attempts can trigger a block against your client address.&lt;/p&gt;

&lt;p&gt;Logs should record successful logins, failed authentication attempts, and administrative actions performed through &lt;code&gt;sudo&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Recent Ubuntu events can be inspected with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;journalctl &lt;span class="nt"&gt;-u&lt;/span&gt; ssh.service &lt;span class="nt"&gt;--since&lt;/span&gt; &lt;span class="s2"&gt;"30 minutes ago"&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;journalctl &lt;span class="nv"&gt;SYSLOG_IDENTIFIER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;sudo&lt;/span&gt; &lt;span class="nt"&gt;--since&lt;/span&gt; &lt;span class="s2"&gt;"30 minutes ago"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An attacker with root access can alter local logs.&lt;/p&gt;

&lt;p&gt;Keeping a copy on another system makes later incident reconstruction more reliable.&lt;/p&gt;

&lt;p&gt;Verify that the records are actually arriving at the remote logging destination, that clocks on both systems are synchronized through NTP, and that logs are retained for as long as required by your audit or operational policy.&lt;/p&gt;

&lt;p&gt;Administrators of the VPS ideally should not have permission to delete the remote copy of those logs.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Prove That the Hardening Works
&lt;/h2&gt;

&lt;p&gt;An audit report should contain the operating system and OpenSSH versions, the list of included configuration files, the inventory of users and keys, and the configuration values before and after modification.&lt;/p&gt;

&lt;p&gt;When comparing configuration files, include the reason for each change.&lt;/p&gt;

&lt;p&gt;Private keys and second-factor secrets must never be included in the report.&lt;/p&gt;

&lt;p&gt;The collected configuration and test results form the audit evidence that demonstrates whether the access policy is actually enforced.&lt;/p&gt;

&lt;p&gt;Test the following scenarios on the server:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Administrator listed in &lt;code&gt;AllowUsers&lt;/code&gt; with a valid key:&lt;/strong&gt; login succeeds and &lt;code&gt;sudo&lt;/code&gt; works&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The same user using only a password:&lt;/strong&gt; login is rejected&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Root with a valid key:&lt;/strong&gt; login is rejected&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;User outside &lt;code&gt;AllowUsers&lt;/code&gt; with a valid key:&lt;/strong&gt; login is rejected&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Connection from a prohibited network when firewall filtering is configured:&lt;/strong&gt; blocked by the firewall&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rollback to the previous configuration:&lt;/strong&gt; new connections work according to the previous rules&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To test password-based authentication from your own computer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nv"&gt;ControlPath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;none &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nv"&gt;PubkeyAuthentication&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;no &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nv"&gt;PreferredAuthentications&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;password,keyboard-interactive &lt;span class="se"&gt;\&lt;/span&gt;
  admin@SERVER
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This test disables public-key authentication.&lt;/p&gt;

&lt;p&gt;When testing a user outside &lt;code&gt;AllowUsers&lt;/code&gt;, use a valid key. Otherwise, the reason for rejection will remain ambiguous because the connection could simply be failing due to invalid credentials.&lt;/p&gt;

&lt;p&gt;For MFA configurations, test a correct code, an incorrect code, and a missing code.&lt;/p&gt;

&lt;p&gt;Record the date, operator, client address, and corresponding log entry alongside each result.&lt;/p&gt;

&lt;p&gt;A firewall rejection can also be confirmed by checking whether the packet or connection counter for the relevant rule increases.&lt;/p&gt;

&lt;p&gt;Repeat the access verification after system updates and whenever the composition of the administrator team changes.&lt;/p&gt;

&lt;p&gt;A good result is simple: permitted users can log in, prohibited scenarios are rejected, and rollback has already been tested instead of being improvised during an outage.&lt;/p&gt;

&lt;p&gt;Keep the SSH hardening checklist together with the recovery procedure.&lt;/p&gt;

&lt;p&gt;Every exception should have a reason, an owner, and a review date.&lt;/p&gt;

&lt;p&gt;That allows the next administrator to distinguish a legitimate operational requirement from a temporary rule that was added months earlier and forgotten.&lt;/p&gt;

</description>
      <category>ssh</category>
      <category>webdev</category>
      <category>programming</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Cloud Server: How It Differs from a VPS and When to Choose the Cloud</title>
      <dc:creator>Aeza</dc:creator>
      <pubDate>Thu, 10 Sep 2026 12:21:21 +0000</pubDate>
      <link>https://dev.to/aeza__net/cloud-server-how-it-differs-from-a-vps-and-when-to-choose-the-cloud-4i5l</link>
      <guid>https://dev.to/aeza__net/cloud-server-how-it-differs-from-a-vps-and-when-to-choose-the-cloud-4i5l</guid>
      <description>&lt;p&gt;A cloud server often looks like a familiar VPS: you choose CPU, memory, and disk, receive an address, and connect over SSH. The difference becomes noticeable when you need to quickly add servers, replace a failed instance, or attach separate storage. These capabilities should be evaluated before migration.&lt;/p&gt;

&lt;p&gt;Configuring cloud servers involves networking, access permissions, and data storage. On a platform with separate managed services, this may require more work than with a traditional VPS. At the same time, a single machine remains a single point of failure: if it becomes unavailable, the application will also stop responding.&lt;/p&gt;

&lt;h3&gt;
  
  
  How does a cloud server differ from a VPS?
&lt;/h3&gt;

&lt;p&gt;A cloud server usually operates as part of a platform with programmable management of machines, storage, and networks. This makes it easier to create resources and adjust their number according to load.&lt;/p&gt;

&lt;p&gt;VPS services can also provide APIs and hourly billing, so the real differences should be evaluated through the capabilities and conditions of specific services.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Is Hidden Behind the Terms VPS and Cloud
&lt;/h2&gt;

&lt;p&gt;A VPS is a virtual machine with its own operating system and a defined share of resources from a physical server. In cloud environments, such a machine is often called an instance. Both services may use the same virtualization technology.&lt;/p&gt;

&lt;p&gt;A cloud platform combines compute, storage, and networking under common management. Users can request resources themselves and release them when they are no longer needed.&lt;/p&gt;

&lt;p&gt;According to the &lt;a href="https://csrc.nist.gov/pubs/sp/800/145/final" rel="noopener noreferrer"&gt;NIST definition of cloud computing&lt;/a&gt;, cloud platforms are characterized by on-demand self-service, broad network access, resource pooling, rapid elasticity, and measured service. Elasticity here means the ability to increase or decrease allocated resources as workload changes.&lt;/p&gt;

&lt;p&gt;In practice, the boundary between VPS and cloud services is blurred. For example, DigitalOcean provides &lt;a href="https://docs.digitalocean.com/products/droplets/reference/api/" rel="noopener noreferrer"&gt;APIs and autoscaling groups&lt;/a&gt; for its virtual machines.&lt;/p&gt;

&lt;p&gt;The useful question is not simply whether an API exists, but what it can actually do: only reboot an instance, or also create disks, configure networks, and replace machines.&lt;/p&gt;

&lt;p&gt;If infrastructure settings rarely change, automation may remain unused, and cloud server administration may differ very little from operating a regular VPS.&lt;/p&gt;

&lt;p&gt;If a team creates test environments every day, however, an API can eliminate a large amount of repetitive work in the control panel.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Configure a Cloud Server: Images, Disks, and Access
&lt;/h2&gt;

&lt;p&gt;Through an API, software can instruct the platform to create a machine from an image, attach a disk, or assign an address.&lt;/p&gt;

&lt;p&gt;An image contains a prepared operating system, while a startup script installs the application and its dependencies. Images and startup scripts make cloud server deployment reproducible.&lt;/p&gt;

&lt;p&gt;Passwords and other secrets should be stored separately from the shared image and provided to applications with limited permissions.&lt;/p&gt;

&lt;p&gt;Access to platform resources is usually controlled through IAM. It defines permissions for employees and software.&lt;/p&gt;

&lt;p&gt;For example, a backup script does not need permission to delete production servers.&lt;/p&gt;

&lt;p&gt;Resources are also commonly grouped into projects so that teams can manage them more clearly.&lt;/p&gt;

&lt;p&gt;Before selecting storage, determine what remains after the machine itself is deleted.&lt;/p&gt;

&lt;p&gt;A local disk is tied to a physical host, while an attached network volume may continue to exist independently of the instance.&lt;/p&gt;

&lt;p&gt;Whether the volume survives deletion depends on the platform and configuration. For example, Amazon EC2 uses the &lt;a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/preserving-volumes-on-termination.html" rel="noopener noreferrer"&gt;&lt;code&gt;DeleteOnTermination&lt;/code&gt;&lt;/a&gt; attribute to determine whether an EBS volume should be deleted together with an instance.&lt;/p&gt;

&lt;p&gt;Object storage is often better suited for user files. An application accesses it through an API rather than as a normal filesystem disk.&lt;/p&gt;

&lt;p&gt;A database can be placed on a suitable volume or delegated to the provider through a managed database service.&lt;/p&gt;

&lt;h2&gt;
  
  
  When Rapid Scaling Actually Works
&lt;/h2&gt;

&lt;p&gt;Increasing the memory or CPU count of a single machine is called &lt;strong&gt;vertical scaling&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Changing the size of an instance may require downtime, so the exact resize procedure should be checked beforehand.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Horizontal scaling&lt;/strong&gt; means adding additional application instances.&lt;/p&gt;

&lt;p&gt;A load balancer distributes incoming requests among them.&lt;/p&gt;

&lt;p&gt;Every instance must have access to shared data such as user sessions, files, and orders.&lt;/p&gt;

&lt;p&gt;If an online store keeps a shopping cart only in the memory of one machine, a second instance will not see it.&lt;/p&gt;

&lt;h3&gt;
  
  
  When Does Cloud Elasticity Justify the Added Complexity?
&lt;/h3&gt;

&lt;p&gt;Elasticity is useful when load varies significantly and additional application instances can start and begin processing traffic before the system becomes overloaded.&lt;/p&gt;

&lt;p&gt;After demand falls, unnecessary resources can be released to reduce costs.&lt;/p&gt;

&lt;p&gt;If the application cannot be distributed across multiple machines, or if the traffic spike ends before additional instances can start, automatic scaling provides little benefit.&lt;/p&gt;

&lt;p&gt;After a machine is created, the operating system still needs to boot, the application has to start, and frequently used data may need to warm the cache.&lt;/p&gt;

&lt;p&gt;In AWS, for example, &lt;a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-default-instance-warmup.html" rel="noopener noreferrer"&gt;instance warmup&lt;/a&gt; defines a period during which metrics from a newly launched instance are excluded from some autoscaling calculations.&lt;/p&gt;

&lt;p&gt;Readiness to accept traffic should be checked separately through health checks. A load balancer needs a reliable way to determine whether the application is actually ready.&lt;/p&gt;

&lt;p&gt;The scaling metric should represent a real shortage of capacity.&lt;/p&gt;

&lt;p&gt;For background workers, the &lt;a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-using-sqs-queue.html" rel="noopener noreferrer"&gt;number of queued jobs per instance&lt;/a&gt; can be useful.&lt;/p&gt;

&lt;p&gt;For compute-heavy workloads, CPU utilization may be appropriate.&lt;/p&gt;

&lt;p&gt;Launching from a prepared image can reduce startup time but does not guarantee that a new server will appear immediately.&lt;/p&gt;

&lt;p&gt;Deployment may still fail because of account quotas or insufficient capacity in a selected zone.&lt;/p&gt;

&lt;p&gt;Maximum instance counts limit spending, while a pre-launched reserve can help absorb short spikes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why One Machine Can Still Fail
&lt;/h2&gt;

&lt;p&gt;During a restart after failure, a service may remain unavailable.&lt;/p&gt;

&lt;p&gt;To continue serving traffic after losing one machine, requests must be accepted by other instances.&lt;/p&gt;

&lt;p&gt;To protect against failure of an entire location, instances can be deployed across different availability zones.&lt;/p&gt;

&lt;p&gt;A zone may contain one or more data centers, but the actual independence of power and networking should be verified with the provider.&lt;/p&gt;

&lt;p&gt;For example, a web application can use a load balancer in front of two application instances placed in separate zones.&lt;/p&gt;

&lt;p&gt;The application uses a shared database, with a primary instance in one zone and a standby in another.&lt;/p&gt;

&lt;p&gt;User files are stored in object storage replicated across multiple zones.&lt;/p&gt;

&lt;p&gt;The load balancer itself must also tolerate a zone failure, and the surviving zone must have enough capacity to handle the full workload.&lt;/p&gt;

&lt;p&gt;Storage architecture has to follow the same logic.&lt;/p&gt;

&lt;p&gt;A network volume may be available only inside one zone. Moving database operations to another zone requires replication and failover.&lt;/p&gt;

&lt;p&gt;For example, Amazon RDS provides &lt;a href="https://aws.amazon.com/rds/features/multi-az/" rel="noopener noreferrer"&gt;Multi-AZ deployments&lt;/a&gt; for this purpose.&lt;/p&gt;

&lt;p&gt;Backups remain necessary even when replicas exist, because accidental deletion or logical corruption can be replicated to the standby system as well.&lt;/p&gt;

&lt;p&gt;An SLA defines the provider's obligations, including availability guarantees for resources.&lt;/p&gt;

&lt;p&gt;The advertised availability percentage of one instance cannot automatically be applied to the entire website because the application still depends on DNS, networking, databases, and other components.&lt;/p&gt;

&lt;p&gt;Amazon EC2, for example, defines different &lt;a href="https://aws.amazon.com/compute/sla/" rel="noopener noreferrer"&gt;SLA commitments&lt;/a&gt; depending on the deployment model.&lt;/p&gt;

&lt;p&gt;The SLA should therefore be checked for compensation rules, exclusions, and the responsibilities that remain with your own team.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Compare Performance and Test Recovery
&lt;/h2&gt;

&lt;p&gt;Even servers with the same number of virtual CPUs and the same amount of memory can perform differently.&lt;/p&gt;

&lt;p&gt;Performance depends on factors such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;physical CPU model&lt;/li&gt;
&lt;li&gt;available CPU time&lt;/li&gt;
&lt;li&gt;disk performance&lt;/li&gt;
&lt;li&gt;network distance&lt;/li&gt;
&lt;li&gt;burst behavior&lt;/li&gt;
&lt;li&gt;IOPS&lt;/li&gt;
&lt;li&gt;throughput&lt;/li&gt;
&lt;li&gt;latency&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A valid comparison requires the same application version, identical data, and the same request profile.&lt;/p&gt;

&lt;p&gt;The load generator must also be powerful enough to generate the required traffic without becoming the bottleneck itself.&lt;/p&gt;

&lt;p&gt;For a web application, useful metrics include requests per second, error rate, and response time.&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;p50&lt;/strong&gt; median shows the response time under which approximately half of requests complete.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;p95&lt;/strong&gt; and &lt;strong&gt;p99&lt;/strong&gt; represent thresholds under which 95% and 99% of requests complete.&lt;/p&gt;

&lt;p&gt;Results under normal and peak load should be compared with CPU, disk, and database utilization.&lt;/p&gt;

&lt;p&gt;A short benchmark may run entirely inside a temporary burst period.&lt;/p&gt;

&lt;p&gt;Some instance types provide extra performance through accumulated credits. Once those credits are exhausted, performance may return to a lower baseline.&lt;/p&gt;

&lt;p&gt;Storage can be constrained both by IOPS and by data throughput.&lt;/p&gt;

&lt;p&gt;With large operations, the throughput limit may be reached before the IOPS limit. Amazon EBS documents this distinction in its &lt;a href="https://docs.aws.amazon.com/ebs/latest/userguide/ebs-io-characteristics.html" rel="noopener noreferrer"&gt;I/O characteristics&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If the service allows temporary burst performance, the test should continue long enough to measure behavior after the burst ends.&lt;/p&gt;

&lt;p&gt;The next step is to verify whether data survives instance replacement.&lt;/p&gt;

&lt;p&gt;On a test copy of the service, write known data to the database and file storage.&lt;/p&gt;

&lt;p&gt;Then perform three separate operations:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Reboot the operating system.&lt;/li&gt;
&lt;li&gt;Stop and start the machine.&lt;/li&gt;
&lt;li&gt;Replace it with a new instance created from the image.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;After each operation, check the data, addresses, and application availability.&lt;/p&gt;

&lt;p&gt;In EC2, for example, &lt;a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-lifetime.html" rel="noopener noreferrer"&gt;instance store&lt;/a&gt; data can survive a reboot but disappears when the instance is stopped or replaced.&lt;/p&gt;

&lt;p&gt;To test fault tolerance, stop one application instance while the system is under load and verify that the second instance continues serving requests.&lt;/p&gt;

&lt;p&gt;Record recovery time, client errors, and the final state of the data.&lt;/p&gt;

&lt;p&gt;Losing an entire availability zone requires a separate test.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Makes Up the Bill
&lt;/h2&gt;

&lt;p&gt;Hourly billing is convenient for temporary workloads.&lt;/p&gt;

&lt;p&gt;A machine running for the entire month accumulates the full monthly compute cost, and storage and other platform services are added on top.&lt;/p&gt;

&lt;p&gt;A monthly calculation should include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Virtual machines:&lt;/strong&gt; runtime hours for each instance × hourly rate, including reserve capacity&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Disks and copies:&lt;/strong&gt; storage volume, additional IOPS, snapshots, and backups&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Networking:&lt;/strong&gt; outbound traffic, cross-zone traffic, and paid addresses&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Platform services:&lt;/strong&gt; load balancers, databases, log storage, and queries&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Team work:&lt;/strong&gt; migration, support, upgrades, and recovery&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For illustration, assume an arbitrary rate of 10 units per hour.&lt;/p&gt;

&lt;p&gt;One machine running for 30 days, or 720 hours, costs 7,200 units.&lt;/p&gt;

&lt;p&gt;If a second machine is required for four hours on 20 days, it adds 800 units, producing a total compute cost of 8,000 units.&lt;/p&gt;

&lt;p&gt;Two continuously running instances would cost 14,400 units for the same period.&lt;/p&gt;

&lt;p&gt;This example includes only compute. Storage, network traffic, backups, and other services still have to be added.&lt;/p&gt;

&lt;p&gt;Some expenses continue after a machine is stopped.&lt;/p&gt;

&lt;p&gt;In EC2, for example, &lt;a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html" rel="noopener noreferrer"&gt;compute billing stops&lt;/a&gt; for a stopped instance while retained EBS volumes continue generating storage charges.&lt;/p&gt;

&lt;p&gt;Rules for addresses, snapshots, and other retained resources depend on the service.&lt;/p&gt;

&lt;p&gt;Resource limits and automatic cleanup of unused infrastructure should therefore be configured deliberately.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which Workloads Are Better Left on a Fixed VPS?
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;A small website with stable traffic that fits comfortably within one selected plan and can tolerate downtime during recovery.&lt;/li&gt;
&lt;li&gt;A bot or internal tool with predictable load and no requirement to remain available during a machine failure.&lt;/li&gt;
&lt;li&gt;An application that cannot yet operate across multiple instances, as long as a single VPS provides enough resources.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  How to Choose the Right Option for Your Workload
&lt;/h2&gt;

&lt;p&gt;How to create a cloud server is usually explained clearly in the provider's documentation.&lt;/p&gt;

&lt;p&gt;The harder part is preparing an application to run across several machines.&lt;/p&gt;

&lt;p&gt;Shared file storage, session data, monitoring, deployment, and recovery procedures all need to be designed.&lt;/p&gt;

&lt;p&gt;The engineering effort required to build and maintain this architecture should be included in the decision.&lt;/p&gt;

&lt;p&gt;A typical comparison looks like this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Management&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fixed VPS: control panel; API capabilities depend on the service&lt;/li&gt;
&lt;li&gt;Cloud platform: creation and management of related resources through APIs&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Load growth&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fixed VPS: upgrade to a larger plan or add more VPS instances&lt;/li&gt;
&lt;li&gt;Cloud platform: machine groups and scaling policies&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Storage&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fixed VPS: disk usually included in the plan&lt;/li&gt;
&lt;li&gt;Cloud platform: separate volumes, object storage, and managed database services&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Networking&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fixed VPS: addresses and networking options included in the selected plan&lt;/li&gt;
&lt;li&gt;Cloud platform: private networks, load balancers, and availability zones&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Availability&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fixed VPS: recovery according to the service conditions&lt;/li&gt;
&lt;li&gt;Cloud platform: multiple zones when the application architecture supports them&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Costs&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fixed VPS: often a fixed monthly amount plus optional extras&lt;/li&gt;
&lt;li&gt;Cloud platform: accounting for multiple independently consumed resources&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An online store with occasional advertising campaigns may be fine with temporarily increasing VPS capacity before a promotion.&lt;/p&gt;

&lt;p&gt;If load fluctuates frequently and unpredictably, automatic instance creation may be more useful, provided the application is designed for it.&lt;/p&gt;

&lt;p&gt;Cloud infrastructure can also be useful under constant load when a team needs managed databases or more convenient recovery mechanisms.&lt;/p&gt;

&lt;p&gt;Several VPS instances behind a load balancer can also serve a distributed application.&lt;/p&gt;

&lt;p&gt;Both a cloud platform and a multi-VPS architecture still require operational work.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Test the Idea Before a Full Migration
&lt;/h2&gt;

&lt;p&gt;A pilot migration is easier to start with a component that does not contain unique local data.&lt;/p&gt;

&lt;p&gt;For example, an image-processing worker can read source files from shared storage and write results to another location.&lt;/p&gt;

&lt;p&gt;Such a component is easier to recreate in the cloud or move back to the VPS.&lt;/p&gt;

&lt;p&gt;Moving the main database during the first stage makes rollback much more difficult.&lt;/p&gt;

&lt;p&gt;Before launching the pilot, define limits for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;response time&lt;/li&gt;
&lt;li&gt;error rate&lt;/li&gt;
&lt;li&gt;recovery time&lt;/li&gt;
&lt;li&gt;monthly budget&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Comparing p95 latency against the target threshold shows whether the configuration is suitable under a real workload.&lt;/p&gt;

&lt;p&gt;Costs should be projected for an entire month, including traffic peaks, even if the pilot itself lasts only one week.&lt;/p&gt;

&lt;p&gt;The pilot plan should record:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;region and availability zones&lt;/li&gt;
&lt;li&gt;instance types&lt;/li&gt;
&lt;li&gt;disk types&lt;/li&gt;
&lt;li&gt;operating system version&lt;/li&gt;
&lt;li&gt;application version&lt;/li&gt;
&lt;li&gt;machine image&lt;/li&gt;
&lt;li&gt;startup procedure&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Logs of resource creation and deletion, request results, and billing exports make the test reproducible.&lt;/p&gt;

&lt;p&gt;You should also write an operational instruction for another team member: how to access the cloud server, where to request permissions, and how to recover the component without the original operator being present.&lt;/p&gt;

&lt;p&gt;At the end of the experiment, return the application to its previous deployment.&lt;/p&gt;

&lt;p&gt;Required data must remain intact and processing must continue correctly.&lt;/p&gt;

&lt;p&gt;A worker may receive the same task more than once after a failure. For example, Amazon SQS standard queues use an &lt;a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html" rel="noopener noreferrer"&gt;at-least-once delivery model&lt;/a&gt;, so processing should be designed to avoid duplicate side effects.&lt;/p&gt;

&lt;p&gt;The experiment should be stopped if it causes data loss, exceeds the defined budget, or requires constant manual intervention.&lt;/p&gt;

&lt;p&gt;After the pilot, remove unused disks, addresses, snapshots, and other resources that can continue generating charges.&lt;/p&gt;

&lt;p&gt;Finally, compare the actual benefit with the cost of migration and ongoing operations.&lt;/p&gt;

&lt;p&gt;Cloud infrastructure pays off when the team uses it as a platform: provisioning resources through APIs, running applications across multiple zones, delegating part of the operational work to managed services, and recovering systems through documented procedures.&lt;/p&gt;

&lt;p&gt;A single virtual machine purchased from a cloud platform instead of a VPS remains a single machine with the same basic point of failure and may simply produce a more complicated bill.&lt;/p&gt;

&lt;p&gt;If a cloud server does not solve any current problem, a working VPS can stay where it is.&lt;/p&gt;

&lt;p&gt;Migration can be reconsidered when there is a concrete reason: increasing load, stronger availability requirements, or too much manual operational work.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>programming</category>
      <category>tutorial</category>
      <category>python</category>
    </item>
    <item>
      <title>Why Your Minecraft Server Lags at 5 GHz: Single-Core Performance, MSPT, and Stable TPS</title>
      <dc:creator>Aeza</dc:creator>
      <pubDate>Mon, 07 Sep 2026 14:44:48 +0000</pubDate>
      <link>https://dev.to/aeza__net/why-your-minecraft-server-lags-at-5-ghz-single-core-performance-mspt-and-stable-tps-5fa9</link>
      <guid>https://dev.to/aeza__net/why-your-minecraft-server-lags-at-5-ghz-single-core-performance-mspt-and-stable-tps-5fa9</guid>
      <description>&lt;p&gt;A CPU advertised at 5 GHz can still run a laggy Minecraft server. Clock speed matters, but it is only one part of the performance equation.&lt;/p&gt;

&lt;p&gt;Stable tick processing also depends on instructions per clock, cache and memory latency, sustained boost behavior, Java garbage collection, server software, plugins, mods, chunk activity, and contention on the host.&lt;/p&gt;

&lt;p&gt;The practical question is not, “Which processor has the highest GHz?”&lt;/p&gt;

&lt;p&gt;It is: &lt;strong&gt;Can the server complete its critical tick-processing work within the available time budget under sustained, representative load?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For a server targeting 20 ticks per second, the nominal budget is 50 milliseconds per tick. When processing regularly exceeds that budget, the server cannot maintain 20 TPS.&lt;/p&gt;

&lt;p&gt;This article explains how to diagnose Minecraft server lag, interpret MSPT, and compare CPUs without relying on advertised boost clocks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why single-core performance matters for a Minecraft server
&lt;/h2&gt;

&lt;p&gt;In conventional &lt;a href="//aeza.net"&gt;Minecraft Java server&lt;/a&gt; implementations, much of the simulation is coordinated by the main server thread.&lt;/p&gt;

&lt;p&gt;During a tick, the server processes world state: entities, scheduled block updates, plugin or mod callbacks, and other required tasks. The exact execution order depends on the implementation and version.&lt;/p&gt;

&lt;p&gt;If the critical work takes too long, the server falls behind.&lt;/p&gt;

&lt;p&gt;Additional cores are still useful. Networking, packet compression, garbage collection, chunk-related work, and explicitly asynchronous tasks can run on other threads. Extra CPU capacity also reduces competition between the main thread and background activity.&lt;/p&gt;

&lt;p&gt;However, additional cores do not automatically shorten a serialized critical path. Doubling the number of cores will not necessarily halve tick time.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Single-core performance matters when the critical thread is computation-bound. Core count matters when parallel work needs additional execution capacity.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This distinction is implementation-dependent. Folia, for example, uses regionized multithreading rather than one conventional world-wide tick thread. Independent regions can run concurrently, but individual busy regions can still become bottlenecks.&lt;/p&gt;

&lt;p&gt;See the &lt;a href="https://docs.papermc.io/" rel="noopener noreferrer"&gt;PaperMC documentation&lt;/a&gt; for the distinction between Paper and Folia.&lt;/p&gt;

&lt;h2&gt;
  
  
  TPS vs. MSPT: what should you measure?
&lt;/h2&gt;

&lt;p&gt;These metrics answer different questions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;TPS ticks per second:&lt;/strong&gt; Is the server maintaining its target simulation rate?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MSPT milliseconds per tick:&lt;/strong&gt; How long does tick processing take?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At the standard 20 TPS target:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1,000 milliseconds / 20 ticks = 50 milliseconds per tick
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A lightly loaded server does not need to spend the full 50 ms processing each tick. It can finish its work early and wait until the next scheduled tick.&lt;/p&gt;

&lt;p&gt;A server that regularly requires more than 50 ms cannot sustain that target indefinitely.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why average MSPT is not enough
&lt;/h3&gt;

&lt;p&gt;Averages can hide intermittent stalls.&lt;/p&gt;

&lt;p&gt;For example, normal ticks may finish quickly while occasional chunk loads, plugin operations, or garbage-collection pauses produce much longer delays.&lt;/p&gt;

&lt;p&gt;Record the distribution, not just its mean:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Median MSPT.&lt;/li&gt;
&lt;li&gt;p95 MSPT.&lt;/li&gt;
&lt;li&gt;p99 MSPT.&lt;/li&gt;
&lt;li&gt;Maximum observed tick duration.&lt;/li&gt;
&lt;li&gt;The proportion of ticks exceeding 50 ms.&lt;/li&gt;
&lt;li&gt;TPS over the same measurement window.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The p99 is the tick duration at or below which approximately 99% of observations fall. It is not the maximum and does not describe every outlier.&lt;/p&gt;

&lt;p&gt;Also record how long the test ran. A percentile from a short, quiet sample is not equivalent to the same percentile measured during a sustained production workload.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stable TPS tells you the server is keeping pace. Tick-time percentiles reveal how consistently it is doing so.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why identical GHz can produce different server performance
&lt;/h2&gt;

&lt;p&gt;Clock frequency measures cycles per second. It does not measure useful work completed per cycle.&lt;/p&gt;

&lt;p&gt;Two processors running at the same frequency can produce different Minecraft tick times because of differences in:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Instructions per clock, or IPC.&lt;/li&gt;
&lt;li&gt;Branch prediction.&lt;/li&gt;
&lt;li&gt;Cache capacity and latency.&lt;/li&gt;
&lt;li&gt;Memory access latency.&lt;/li&gt;
&lt;li&gt;Execution resources.&lt;/li&gt;
&lt;li&gt;Microarchitecture.&lt;/li&gt;
&lt;li&gt;Power and thermal behavior.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Minecraft server workloads often traverse interconnected objects, execute conditional logic, and access data with imperfect locality.&lt;/p&gt;

&lt;p&gt;A cache miss can make the processor wait for data. A branch misprediction can discard speculative work. Higher frequency does not eliminate either problem.&lt;/p&gt;

&lt;p&gt;IPC is also workload-dependent. A processor does not have one universal IPC value that predicts performance across every game, plugin, or server configuration.&lt;/p&gt;

&lt;h3&gt;
  
  
  Your world changes the workload
&lt;/h3&gt;

&lt;p&gt;An empty world is not a substitute for a production environment.&lt;/p&gt;

&lt;p&gt;Entities, farms, hoppers, redstone, custom mobs, plugins, and player distribution all change the amount and shape of the work.&lt;/p&gt;

&lt;p&gt;A generic single-thread benchmark can help shortlist processors. It cannot establish which CPU will deliver the best p99 MSPT on your actual server.&lt;/p&gt;

&lt;h2&gt;
  
  
  Advertised boost clock vs. sustained effective clock
&lt;/h2&gt;

&lt;p&gt;The maximum boost frequency in a specification is a peak operating condition, not a promise that the CPU will maintain that frequency throughout a long server workload.&lt;/p&gt;

&lt;p&gt;Sustained behavior depends on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Temperature and cooling.&lt;/li&gt;
&lt;li&gt;Power limits.&lt;/li&gt;
&lt;li&gt;Firmware configuration.&lt;/li&gt;
&lt;li&gt;CPU frequency-management policy.&lt;/li&gt;
&lt;li&gt;The number of active cores.&lt;/li&gt;
&lt;li&gt;Other workloads sharing the machine.&lt;/li&gt;
&lt;li&gt;Virtualization and compute-allocation limits.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A short benchmark can finish before the system reaches a stable thermal state. A longer run may reveal reduced frequency or inconsistent CPU availability.&lt;/p&gt;

&lt;p&gt;That is why effective clock and MSPT should be measured during the same interval.&lt;/p&gt;

&lt;h3&gt;
  
  
  What changes on a VPS?
&lt;/h3&gt;

&lt;p&gt;Inside a virtual machine, frequency readings and hardware telemetry can be incomplete or misleading.&lt;/p&gt;

&lt;p&gt;The guest may report a nominal frequency without exposing the host’s actual thermal limits, power constraints, or scheduling decisions.&lt;/p&gt;

&lt;p&gt;For a Minecraft VPS, inspect more than the advertised CPU model:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Are vCPUs shared or dedicated?&lt;/li&gt;
&lt;li&gt;Is compute capacity burstable or sustained?&lt;/li&gt;
&lt;li&gt;What limits apply to CPU use?&lt;/li&gt;
&lt;li&gt;Is host contention visible through steal-time metrics?&lt;/li&gt;
&lt;li&gt;Are repeated performance measurements consistent?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Where available, CPU steal time indicates time during which the virtual CPU was not executing because the hypervisor was servicing other work. Missing or low reported steal time does not prove that every other resource limit is absent.&lt;/p&gt;

&lt;h2&gt;
  
  
  High MSPT does not automatically mean a slow CPU
&lt;/h2&gt;

&lt;p&gt;The server thread may be executing code, or it may be waiting.&lt;/p&gt;

&lt;p&gt;Those situations require different fixes.&lt;/p&gt;

&lt;p&gt;Possible causes of Minecraft server lag include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Expensive synchronous plugin or mod code.&lt;/li&gt;
&lt;li&gt;Entity AI and pathfinding.&lt;/li&gt;
&lt;li&gt;Large numbers of block, hopper, or redstone updates.&lt;/li&gt;
&lt;li&gt;Chunk generation and loading.&lt;/li&gt;
&lt;li&gt;Synchronous disk operations.&lt;/li&gt;
&lt;li&gt;Lock contention.&lt;/li&gt;
&lt;li&gt;Java garbage-collection pauses.&lt;/li&gt;
&lt;li&gt;Memory pressure or swapping.&lt;/li&gt;
&lt;li&gt;CPU throttling.&lt;/li&gt;
&lt;li&gt;Host-level scheduling contention.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A faster CPU can help a computation-bound workload. It may do little for a thread blocked on storage, waiting for another task, or paused by garbage collection.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Identify the bottleneck before buying hardware.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How to investigate server lag with spark
&lt;/h2&gt;

&lt;p&gt;The spark profiler helps connect performance problems to the code executing during a workload.&lt;/p&gt;

&lt;p&gt;For a conventional Minecraft server, the &lt;code&gt;Server thread&lt;/code&gt; is usually the first thread to inspect.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://spark.lucko.me/docs/Using-the-viewer" rel="noopener noreferrer"&gt;spark viewer documentation&lt;/a&gt; explains how to navigate threads and call trees.&lt;/p&gt;

&lt;p&gt;Capture a profile while the problem is happening. A quiet-period profile will not explain a busy-period slowdown.&lt;/p&gt;

&lt;p&gt;Useful commands include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/spark health
/spark profiler start
/spark profiler stop
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a background profiler is already running, follow the installed version’s instructions for opening or stopping that session.&lt;/p&gt;

&lt;p&gt;To investigate slow ticks specifically, spark supports:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/spark profiler start --only-ticks-over 50
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This filters profiling to ticks exceeding the specified duration. It is useful for investigating spikes, but a filtered profile should not be treated as a complete description of normal server activity.&lt;/p&gt;

&lt;p&gt;Command behavior and availability depend on the installed version and platform. Refer to the &lt;a href="https://spark.lucko.me/docs/Command-Usage" rel="noopener noreferrer"&gt;official spark command reference&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Keep the profile’s context
&lt;/h3&gt;

&lt;p&gt;A profile without workload information is difficult to compare.&lt;/p&gt;

&lt;p&gt;Record:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Server implementation and exact build.&lt;/li&gt;
&lt;li&gt;Minecraft version.&lt;/li&gt;
&lt;li&gt;Java version.&lt;/li&gt;
&lt;li&gt;JVM flags and heap configuration.&lt;/li&gt;
&lt;li&gt;World snapshot identifier.&lt;/li&gt;
&lt;li&gt;Plugin or mod versions.&lt;/li&gt;
&lt;li&gt;Player count.&lt;/li&gt;
&lt;li&gt;View distance.&lt;/li&gt;
&lt;li&gt;Simulation distance.&lt;/li&gt;
&lt;li&gt;Test actions and duration.&lt;/li&gt;
&lt;li&gt;Relevant garbage-collection logs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Collect MSPT statistics, profiles, and GC events over matching time windows.&lt;/p&gt;

&lt;p&gt;A call tree shows where samples were attributed. It does not automatically prove that every delay was active CPU execution. Interpret it alongside the profiler mode, thread state, and operating-system telemetry.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to benchmark a Minecraft server CPU properly
&lt;/h2&gt;

&lt;p&gt;A meaningful comparison changes the CPU or hosting environment while keeping the workload as consistent as possible.&lt;/p&gt;

&lt;p&gt;Use identical copies of the world and match the server build, Java runtime, heap size, plugins, mods, and configuration.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Define the target before testing
&lt;/h3&gt;

&lt;p&gt;“Runs Minecraft well” is not a measurable requirement.&lt;/p&gt;

&lt;p&gt;An example acceptance target could be:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Maintain 20 TPS with p99 MSPT below 45 ms during the normal-load scenario, without sustained tick backlog during the heavy-load scenario.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The 45 ms threshold is an example, not a universal standard. Choose a target that matches your service requirements and leaves appropriate headroom.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Separate workload scenarios
&lt;/h3&gt;

&lt;p&gt;Test these independently:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Idle:&lt;/strong&gt; Background activity and loaded chunks without players.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Normal load:&lt;/strong&gt; Representative players, farms, entities, and plugin activity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chunk generation:&lt;/strong&gt; A controlled workload involving new terrain.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Heavy event:&lt;/strong&gt; The largest legitimate event expected in production.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Do not combine all scenarios into one average. Each can expose a different bottleneck.&lt;/p&gt;

&lt;p&gt;Player count alone is insufficient. Fifty players exploring separate areas may create a different workload from fifty players gathered around one complex farm.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Warm up the server
&lt;/h3&gt;

&lt;p&gt;Allow the JVM and hardware to reach a reasonably stable operating state.&lt;/p&gt;

&lt;p&gt;Warm-up matters because JIT compilation, cache state, temperature, and power behavior can affect early results.&lt;/p&gt;

&lt;p&gt;Use the same warm-up policy for every candidate.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Record measurements together
&lt;/h3&gt;

&lt;p&gt;During each measured interval, capture:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tick-time distribution.&lt;/li&gt;
&lt;li&gt;TPS.&lt;/li&gt;
&lt;li&gt;Main-thread profile.&lt;/li&gt;
&lt;li&gt;GC events.&lt;/li&gt;
&lt;li&gt;CPU utilization.&lt;/li&gt;
&lt;li&gt;Effective clock, where available.&lt;/li&gt;
&lt;li&gt;Temperature and power-limit information, where available.&lt;/li&gt;
&lt;li&gt;CPU steal time on virtual machines.&lt;/li&gt;
&lt;li&gt;Storage latency when I/O is suspected.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On Linux, the following commands can help record machine configuration:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;lscpu &lt;span class="nt"&gt;-e&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;CPU,CORE,SOCKET,NODE,ONLINE &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; topology.txt
pgrep &lt;span class="nt"&gt;-a&lt;/span&gt; &lt;span class="nt"&gt;-x&lt;/span&gt; java
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Identify the correct server process before setting its PID:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;JAVA_PID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;12345  &lt;span class="c"&gt;# Replace with the actual server process ID&lt;/span&gt;

taskset &lt;span class="nt"&gt;-pc&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$JAVA_PID&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; affinity.txt
cpupower frequency-info &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; governor.txt
&lt;span class="nb"&gt;sudo timeout &lt;/span&gt;180 turbostat &lt;span class="nt"&gt;--interval&lt;/span&gt; 1 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; turbo.txt 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In this form, &lt;code&gt;taskset -pc&lt;/code&gt; reports the affinity mask without changing it.&lt;/p&gt;

&lt;p&gt;These commands require the relevant utilities to be installed. &lt;code&gt;turbostat&lt;/code&gt; needs supported hardware and appropriate permissions. Useful telemetry may be unavailable inside a VPS.&lt;/p&gt;

&lt;p&gt;Do not interpret missing sensor output as evidence that thermal or power limits are irrelevant.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Repeat the tests
&lt;/h3&gt;

&lt;p&gt;Use at least three warmed-up repetitions as a starting point, with longer or additional runs when variability remains high.&lt;/p&gt;

&lt;p&gt;Alternate the order of candidates where practical.&lt;/p&gt;

&lt;p&gt;Restore the same initial world state when a scenario modifies it. Otherwise, the second candidate may receive a materially different workload.&lt;/p&gt;

&lt;p&gt;Preserve raw results, not only screenshots.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Compare latency tails and consistency
&lt;/h3&gt;

&lt;p&gt;A CPU with lower average MSPT may still produce worse p99 latency.&lt;/p&gt;

&lt;p&gt;Compare:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Performance in each scenario.&lt;/li&gt;
&lt;li&gt;Run-to-run variation.&lt;/li&gt;
&lt;li&gt;Frequency of over-budget ticks.&lt;/li&gt;
&lt;li&gt;Severity and duration of stalls.&lt;/li&gt;
&lt;li&gt;Cost and available resource headroom.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not claim a meaningful improvement from a small difference that falls within normal measurement variability.&lt;/p&gt;

&lt;h2&gt;
  
  
  How many CPU cores does a Minecraft server need?
&lt;/h2&gt;

&lt;p&gt;There is no universal answer.&lt;/p&gt;

&lt;p&gt;A conventional server benefits from strong sustained per-core performance and enough additional capacity for background work.&lt;/p&gt;

&lt;p&gt;A balanced configuration provides:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fast execution of the critical tick-processing work.&lt;/li&gt;
&lt;li&gt;Additional cores for parallel and background tasks.&lt;/li&gt;
&lt;li&gt;Sufficient RAM without host memory pressure.&lt;/li&gt;
&lt;li&gt;Consistent storage latency.&lt;/li&gt;
&lt;li&gt;Cooling and power limits suitable for sustained load.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;More cores can also help when running multiple independent server instances on the same machine.&lt;/p&gt;

&lt;p&gt;However, a large core count does not guarantee that one overloaded main thread will run faster.&lt;/p&gt;

&lt;p&gt;For shared hosting, the allocation policy matters as much as the advertised vCPU count.&lt;/p&gt;

&lt;h2&gt;
  
  
  When a CPU upgrade will not fix the problem
&lt;/h2&gt;

&lt;p&gt;A CPU upgrade is most likely to help when profiling shows sustained computation on the critical thread.&lt;/p&gt;

&lt;p&gt;Other findings suggest different actions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Blocking plugin operations:&lt;/strong&gt; Inspect or replace the relevant plugin.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Slow chunk generation:&lt;/strong&gt; Evaluate pre-generation and workload limits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Storage stalls:&lt;/strong&gt; Investigate I/O latency and synchronous operations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GC pauses:&lt;/strong&gt; Inspect allocation behavior, heap configuration, and GC logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lock contention:&lt;/strong&gt; Investigate synchronization and plugin architecture.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Host contention:&lt;/strong&gt; Compare compute guarantees and repeated measurements.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;High network latency with healthy ticks:&lt;/strong&gt; Investigate the network separately.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are diagnostic directions, not automatic fixes. Confirm the cause with evidence.&lt;/p&gt;

&lt;p&gt;More RAM is not a universal solution either. It helps when insufficient memory is part of the problem; it does not directly accelerate a computation-bound server thread.&lt;/p&gt;

&lt;h2&gt;
  
  
  Choosing the best CPU for your Minecraft server
&lt;/h2&gt;

&lt;p&gt;Use single-thread benchmarks to narrow the shortlist, then test candidates on a representative world.&lt;/p&gt;

&lt;p&gt;For published comparisons, look for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Exact software versions.&lt;/li&gt;
&lt;li&gt;World and workload details.&lt;/li&gt;
&lt;li&gt;Player behavior, not only player count.&lt;/li&gt;
&lt;li&gt;Java and memory configuration.&lt;/li&gt;
&lt;li&gt;Test duration and repetitions.&lt;/li&gt;
&lt;li&gt;Thermal conditions.&lt;/li&gt;
&lt;li&gt;Tick-time distributions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A screenshot showing 20 TPS on an unspecified world is not a meaningful capacity benchmark.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The best CPU for a Minecraft server is the one that delivers consistent tick times on your workload, within your budget not necessarily the one with the highest advertised GHz.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Does Minecraft use only one CPU core?
&lt;/h3&gt;

&lt;p&gt;No. Minecraft servers can use multiple threads for networking, garbage collection, chunk-related work, and background tasks.&lt;/p&gt;

&lt;p&gt;However, conventional implementations coordinate substantial simulation work through a main thread. Regionized implementations such as Folia use a different model.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is a 5 GHz CPU good for a Minecraft server?
&lt;/h3&gt;

&lt;p&gt;Potentially, but frequency alone is insufficient.&lt;/p&gt;

&lt;p&gt;Architecture, cache, memory behavior, sustained clock speed, compute allocation, and the workload all affect performance.&lt;/p&gt;

&lt;p&gt;Measure the server instead of judging only the specification.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is a good MSPT?
&lt;/h3&gt;

&lt;p&gt;For a 20 TPS target, the nominal budget is 50 ms per tick.&lt;/p&gt;

&lt;p&gt;Lower processing times provide headroom, but examine the distribution as well as the average. Occasional long ticks can still cause noticeable disruption.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does my server lag with low overall CPU usage?
&lt;/h3&gt;

&lt;p&gt;Aggregate CPU utilization averages activity across the available processors.&lt;/p&gt;

&lt;p&gt;One critical thread can saturate a logical CPU while much of the machine remains idle. Alternatively, the thread may be waiting rather than executing.&lt;/p&gt;

&lt;p&gt;Inspect per-thread activity and profile the workload.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should I measure TPS or MSPT?
&lt;/h3&gt;

&lt;p&gt;Use both.&lt;/p&gt;

&lt;p&gt;TPS shows whether simulation is keeping pace. MSPT describes processing time, while profiling and supporting telemetry help explain the cause of slow ticks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Clock speed contributes to Minecraft server performance, but it is not a complete performance metric.&lt;/p&gt;

&lt;p&gt;Reliable diagnosis requires a representative world, repeatable scenarios, warmed-up measurements, tick-time percentiles, and profiles captured during the actual problem.&lt;/p&gt;

&lt;p&gt;First determine whether the critical thread is computing, waiting, or being paused. Then decide whether the right change is a faster CPU, better resource guarantees, a plugin fix, memory tuning, or storage work.&lt;/p&gt;

&lt;p&gt;That approach avoids paying for a higher GHz number while leaving the real bottleneck untouched.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://spark.lucko.me/docs/Using-the-viewer" rel="noopener noreferrer"&gt;spark: Using the viewer&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://spark.lucko.me/docs/Command-Usage" rel="noopener noreferrer"&gt;spark: Command usage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://spark.lucko.me/docs/Developer-API" rel="noopener noreferrer"&gt;spark: Developer API&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.papermc.io/" rel="noopener noreferrer"&gt;PaperMC documentation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>minecraft</category>
      <category>java</category>
      <category>performance</category>
      <category>devops</category>
    </item>
    <item>
      <title>Ollama vs vLLM for Concurrent Inference: How to Choose the Right Server</title>
      <dc:creator>Aeza</dc:creator>
      <pubDate>Thu, 03 Sep 2026 14:13:06 +0000</pubDate>
      <link>https://dev.to/aeza__net/ollama-vs-vllm-for-concurrent-inference-how-to-choose-the-right-server-3o5p</link>
      <guid>https://dev.to/aeza__net/ollama-vs-vllm-for-concurrent-inference-how-to-choose-the-right-server-3o5p</guid>
      <description>&lt;p&gt;Choosing between Ollama and vLLM is not simply a matter of asking:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which inference engine is faster?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The answer depends heavily on the workload.&lt;/p&gt;

&lt;p&gt;A personal chat interface with one active user has very different requirements from a multi-user API handling dozens of concurrent generations.&lt;/p&gt;

&lt;p&gt;For one workload, simple model management may matter more than maximum throughput.&lt;/p&gt;

&lt;p&gt;For another, the deciding factors may be:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;continuous batching&lt;/li&gt;
&lt;li&gt;queue behavior&lt;/li&gt;
&lt;li&gt;KV-cache utilization&lt;/li&gt;
&lt;li&gt;TTFT&lt;/li&gt;
&lt;li&gt;p99 latency&lt;/li&gt;
&lt;li&gt;GPU saturation&lt;/li&gt;
&lt;li&gt;overload handling&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That means a useful Ollama vs vLLM comparison cannot be reduced to a single &lt;code&gt;tokens/s&lt;/code&gt; number.&lt;/p&gt;

&lt;p&gt;In this article, we'll build a practical benchmark methodology for comparing the two under real concurrent load.&lt;/p&gt;

&lt;p&gt;By the end, you'll know:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;when vLLM's batching can provide a measurable advantage&lt;/li&gt;
&lt;li&gt;why Ollama may still be the better choice for smaller workloads&lt;/li&gt;
&lt;li&gt;how to build a fair concurrency sweep&lt;/li&gt;
&lt;li&gt;which latency and throughput metrics actually matter&lt;/li&gt;
&lt;li&gt;how to detect queueing and saturation&lt;/li&gt;
&lt;li&gt;how to choose the server that stays inside your SLO&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  The Short Answer
&lt;/h2&gt;

&lt;p&gt;There is no universal concurrency level at which vLLM suddenly becomes better than Ollama.&lt;/p&gt;

&lt;p&gt;vLLM begins to gain an advantage when enough requests are active or waiting for its scheduler to form useful batches while the GPU still has sufficient compute and memory capacity.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Low concurrency
      ↓
Little opportunity for batching
      ↓
Ollama may be completely sufficient

Higher concurrency
      ↓
More simultaneous requests
      ↓
Continuous batching becomes useful
      ↓
vLLM can improve aggregate throughput

Too much concurrency
      ↓
Queue grows
      ↓
TTFT and p99 increase
      ↓
OOM / timeout / rejected requests
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The real question is therefore not:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which engine reaches the highest tokens per second?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which engine delivers the required throughput while keeping latency, errors, memory usage, and operational complexity inside acceptable limits?&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Start with the Workload, Not the Inference Engine
&lt;/h2&gt;

&lt;p&gt;Before testing either server, define what the service actually needs to handle.&lt;/p&gt;

&lt;p&gt;Three common scenarios illustrate why this matters.&lt;/p&gt;

&lt;h3&gt;
  
  
  Personal Chat
&lt;/h3&gt;

&lt;p&gt;A personal chat usually has:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;one active user&lt;/li&gt;
&lt;li&gt;low concurrency&lt;/li&gt;
&lt;li&gt;occasional model loading&lt;/li&gt;
&lt;li&gt;relatively tolerant startup latency&lt;/li&gt;
&lt;li&gt;no large server-side request queue&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In this environment, maximum aggregate GPU throughput may not matter very much.&lt;/p&gt;

&lt;p&gt;Operational simplicity often matters more.&lt;/p&gt;

&lt;h3&gt;
  
  
  Small Internal API
&lt;/h3&gt;

&lt;p&gt;An internal API may receive:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;several simultaneous users&lt;/li&gt;
&lt;li&gt;short request bursts&lt;/li&gt;
&lt;li&gt;varying context lengths&lt;/li&gt;
&lt;li&gt;occasional concurrency spikes&lt;/li&gt;
&lt;li&gt;moderate latency requirements&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both Ollama and vLLM may fit this workload.&lt;/p&gt;

&lt;p&gt;The correct choice depends on the actual traffic curve.&lt;/p&gt;

&lt;h3&gt;
  
  
  High-Concurrency Inference Service
&lt;/h3&gt;

&lt;p&gt;A public or heavily used internal service may have:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;many simultaneous requests&lt;/li&gt;
&lt;li&gt;sustained queues&lt;/li&gt;
&lt;li&gt;strict TTFT targets&lt;/li&gt;
&lt;li&gt;p95/p99 latency requirements&lt;/li&gt;
&lt;li&gt;high GPU utilization targets&lt;/li&gt;
&lt;li&gt;overload conditions that must be handled predictably&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is where continuous batching and queue-aware scheduling become much more important.&lt;/p&gt;




&lt;h2&gt;
  
  
  Record the Workload Before Benchmarking
&lt;/h2&gt;

&lt;p&gt;At minimum, record:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Input tokens&lt;/strong&gt; — determine prefill cost.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Output tokens&lt;/strong&gt; — determine how long a request occupies generation resources.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Average concurrency&lt;/strong&gt; — represents normal load.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Peak concurrency&lt;/strong&gt; — reveals saturation behavior.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Streaming ratio&lt;/strong&gt; — affects user-perceived latency.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Context length&lt;/strong&gt; — directly affects KV-cache consumption.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Request bursts&lt;/strong&gt; — determine queue behavior.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Target SLO&lt;/strong&gt; — defines when performance becomes unacceptable.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Long contexts consume more KV-cache.&lt;/p&gt;

&lt;p&gt;Long generations keep execution resources occupied for longer.&lt;/p&gt;

&lt;p&gt;Average concurrency can also hide dangerous workload patterns.&lt;/p&gt;

&lt;p&gt;For example, imagine a service that usually has four active requests.&lt;/p&gt;

&lt;p&gt;That sounds moderate.&lt;/p&gt;

&lt;p&gt;But if several long conversations overlap at the same moment, the system may suddenly experience:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;KV-cache pressure
+
longer queue
+
higher TTFT
+
higher p99
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;even though average concurrency still looks reasonable.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Important:&lt;/strong&gt; Higher server throughput does not automatically mean a better user experience.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A configuration can produce more total tokens per second while making individual requests noticeably slower.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why Ollama Is Attractive for Smaller Workloads
&lt;/h2&gt;

&lt;p&gt;Ollama combines several parts of the local model lifecycle into one workflow:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;model downloading&lt;/li&gt;
&lt;li&gt;model storage&lt;/li&gt;
&lt;li&gt;Modelfiles&lt;/li&gt;
&lt;li&gt;configuration&lt;/li&gt;
&lt;li&gt;local HTTP API&lt;/li&gt;
&lt;li&gt;model loading&lt;/li&gt;
&lt;li&gt;model unloading&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a developer or small team, this can make deployment and maintenance significantly easier.&lt;/p&gt;

&lt;p&gt;You can replace a model quickly, modify the system prompt, and control how long a model remains loaded without building a large serving stack.&lt;/p&gt;

&lt;p&gt;Cold-start performance should still be evaluated separately from warm inference.&lt;/p&gt;




&lt;h2&gt;
  
  
  Important Ollama Parameters
&lt;/h2&gt;

&lt;p&gt;In a warmed-up environment, Ollama behavior depends on settings such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OLLAMA_NUM_PARALLEL
OLLAMA_MAX_QUEUE
OLLAMA_MAX_LOADED_MODELS
keep_alive
context size
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Parallel requests increase memory consumption because each active request may require additional context state.&lt;/p&gt;

&lt;p&gt;A configuration that works perfectly with short prompts may behave very differently with longer contexts.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Short context
+
4 parallel requests
=
works normally
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;may become:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Long context
+
4 parallel requests
=
larger memory usage
+
queue growth
+
possible failure
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So do not treat parallelism as an isolated setting.&lt;/p&gt;

&lt;p&gt;Context length matters too.&lt;/p&gt;




&lt;h2&gt;
  
  
  Ollama's Main Advantage
&lt;/h2&gt;

&lt;p&gt;For many workloads, Ollama's strongest advantage is not peak benchmark performance.&lt;/p&gt;

&lt;p&gt;It is operational simplicity.&lt;/p&gt;

&lt;p&gt;That can be extremely valuable for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;personal assistants&lt;/li&gt;
&lt;li&gt;development environments&lt;/li&gt;
&lt;li&gt;prototypes&lt;/li&gt;
&lt;li&gt;small internal APIs&lt;/li&gt;
&lt;li&gt;teams that frequently change models&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the workload does not create enough concurrency for continuous batching to matter, a more complicated serving stack may provide little practical benefit.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why vLLM Benefits from Concurrency
&lt;/h2&gt;

&lt;p&gt;vLLM is designed around server-side inference workloads.&lt;/p&gt;

&lt;p&gt;Its scheduler can continuously adjust active batches as requests arrive and complete.&lt;/p&gt;

&lt;p&gt;Instead of waiting for one static batch to fully finish, the engine can use newly available capacity for other requests.&lt;/p&gt;

&lt;p&gt;This is commonly referred to as &lt;strong&gt;continuous batching&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Request A ────────────────&amp;gt;
Request B      ────────────────&amp;gt;
Request C           ────────────────&amp;gt;

Scheduler continuously updates
the active batch as requests
arrive and finish.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This becomes useful when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;several requests are active&lt;/li&gt;
&lt;li&gt;more requests are waiting&lt;/li&gt;
&lt;li&gt;the GPU supports the required backend&lt;/li&gt;
&lt;li&gt;enough memory remains available&lt;/li&gt;
&lt;li&gt;batching improves accelerator utilization&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;With one short request, there may simply be nothing useful to batch.&lt;/p&gt;




&lt;h2&gt;
  
  
  Important vLLM Parameters
&lt;/h2&gt;

&lt;p&gt;Record the server configuration with every benchmark.&lt;/p&gt;

&lt;p&gt;Important settings include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nt"&gt;--max-num-seqs&lt;/span&gt;
&lt;span class="nt"&gt;--max-model-len&lt;/span&gt;
&lt;span class="nt"&gt;--gpu-memory-utilization&lt;/span&gt;
&lt;span class="nt"&gt;--dtype&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Also record whether chunked prefill is enabled and how it is configured.&lt;/p&gt;

&lt;p&gt;For multi-GPU deployments, record:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nt"&gt;--tensor-parallel-size&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tensor parallelism can increase the effective memory available for larger models, but it also introduces communication between GPUs.&lt;/p&gt;

&lt;p&gt;That communication is not free.&lt;/p&gt;

&lt;p&gt;So:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;More GPUs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;does not automatically mean:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Linear performance scaling
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Monitor vLLM's Server-Side Behavior
&lt;/h2&gt;

&lt;p&gt;In addition to normal application metrics, monitor:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;request queue length&lt;/li&gt;
&lt;li&gt;KV-cache utilization&lt;/li&gt;
&lt;li&gt;GPU memory utilization&lt;/li&gt;
&lt;li&gt;preemption&lt;/li&gt;
&lt;li&gt;rejected requests&lt;/li&gt;
&lt;li&gt;OOM events&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both Ollama and vLLM should also be monitored for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;queue growth&lt;/li&gt;
&lt;li&gt;memory pressure&lt;/li&gt;
&lt;li&gt;timeouts&lt;/li&gt;
&lt;li&gt;failed requests&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The important difference is that vLLM exposes more of the machinery involved in high-concurrency serving.&lt;/p&gt;




&lt;h2&gt;
  
  
  Which Server Is Easier for One User?
&lt;/h2&gt;

&lt;p&gt;For a single user, Ollama will usually be easier to operate.&lt;/p&gt;

&lt;p&gt;The model lifecycle and API are integrated into one workflow.&lt;/p&gt;

&lt;p&gt;vLLM becomes more attractive when you specifically require:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;high concurrent throughput&lt;/li&gt;
&lt;li&gt;continuous batching&lt;/li&gt;
&lt;li&gt;detailed production metrics&lt;/li&gt;
&lt;li&gt;specific GPU backends&lt;/li&gt;
&lt;li&gt;larger request queues&lt;/li&gt;
&lt;li&gt;a workload expected to grow rapidly&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The decision should follow the workload rather than the popularity of either tool.&lt;/p&gt;




&lt;h2&gt;
  
  
  Make the Benchmark Fair
&lt;/h2&gt;

&lt;p&gt;A benchmark only makes sense if both systems are processing equivalent work.&lt;/p&gt;

&lt;p&gt;This sounds obvious.&lt;/p&gt;

&lt;p&gt;In practice, it is easy to get wrong.&lt;/p&gt;

&lt;p&gt;Try to keep the following equivalent:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;model weights&lt;/li&gt;
&lt;li&gt;model architecture&lt;/li&gt;
&lt;li&gt;precision&lt;/li&gt;
&lt;li&gt;quantization&lt;/li&gt;
&lt;li&gt;tokenizer&lt;/li&gt;
&lt;li&gt;chat template&lt;/li&gt;
&lt;li&gt;generation parameters&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Different weight formats may require different inference paths.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GGUF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;safetensors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;do not automatically represent identical runtime behavior.&lt;/p&gt;

&lt;p&gt;If exactly the same model build cannot be used by both engines, document that clearly.&lt;/p&gt;

&lt;p&gt;The benchmark conclusion then applies only to the tested configurations.&lt;/p&gt;




&lt;h2&gt;
  
  
  Record Model Sources and Hashes
&lt;/h2&gt;

&lt;p&gt;For reproducibility, save:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Model name
Model version
Weight source
File hash
Quantization
dtype
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Otherwise, a future benchmark may use slightly different weights and produce results that appear inconsistent.&lt;/p&gt;




&lt;h2&gt;
  
  
  Keep Generation Parameters Identical
&lt;/h2&gt;

&lt;p&gt;Use the same values for parameters such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;max_tokens
temperature
top_p
stop
stream
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For deterministic benchmark workloads, you may use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;temperature: 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But matching generation settings is still not enough.&lt;/p&gt;




&lt;h2&gt;
  
  
  Verify Tokenization and Chat Templates
&lt;/h2&gt;

&lt;p&gt;The same chat payload can produce different token sequences if the two servers use different tokenizers or templates.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"messages"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Explain TCP slow start."&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;may be transformed differently before reaching the model.&lt;/p&gt;

&lt;p&gt;That means two systems may appear to be benchmarking the same request while actually processing different input lengths.&lt;/p&gt;

&lt;p&gt;Compare input-token counts before trusting the results.&lt;/p&gt;

&lt;p&gt;If one server sees:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;312 input tokens
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and another sees:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;356 input tokens
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;the workload is not identical.&lt;/p&gt;




&lt;h2&gt;
  
  
  Record the Benchmark Environment
&lt;/h2&gt;

&lt;p&gt;Every run should include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Inference server version
Model name
Model hash
GPU model
VRAM
CPU
RAM
Operating system
Driver version
Context length
Generation settings
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A result such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8,200 output tokens/s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;has little value without the environment that produced it.&lt;/p&gt;




&lt;h2&gt;
  
  
  Separate Cold and Warm Performance
&lt;/h2&gt;

&lt;p&gt;Cold-start performance and steady-state inference answer different questions.&lt;/p&gt;

&lt;p&gt;Measure them separately.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cold Test
&lt;/h3&gt;

&lt;p&gt;A cold test includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;model loading&lt;/li&gt;
&lt;li&gt;initialization&lt;/li&gt;
&lt;li&gt;first request latency&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This matters for workloads where models are frequently unloaded.&lt;/p&gt;

&lt;h3&gt;
  
  
  Warm Test
&lt;/h3&gt;

&lt;p&gt;A warm test measures:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;steady-state throughput&lt;/li&gt;
&lt;li&gt;TTFT&lt;/li&gt;
&lt;li&gt;token generation&lt;/li&gt;
&lt;li&gt;latency&lt;/li&gt;
&lt;li&gt;concurrency scaling&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not combine cold and warm measurements into one average.&lt;/p&gt;

&lt;p&gt;They describe different behavior.&lt;/p&gt;




&lt;h2&gt;
  
  
  Handle Prompt Caching Consistently
&lt;/h2&gt;

&lt;p&gt;Prompt caching can significantly change benchmark results.&lt;/p&gt;

&lt;p&gt;Either:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;disable prompt caching on both systems
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;use it consistently on both systems
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do not allow one server to reuse cached prompt state while the other processes every prompt from scratch.&lt;/p&gt;




&lt;h2&gt;
  
  
  Make Sure the Load Generator Is Not the Bottleneck
&lt;/h2&gt;

&lt;p&gt;Your client must have enough:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU&lt;/li&gt;
&lt;li&gt;network bandwidth&lt;/li&gt;
&lt;li&gt;connection capacity&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;to generate the intended load.&lt;/p&gt;

&lt;p&gt;Otherwise, you may accidentally benchmark the load generator rather than the inference server.&lt;/p&gt;

&lt;p&gt;Monitor the client machine during heavy tests too.&lt;/p&gt;




&lt;h2&gt;
  
  
  Build a Concurrency Sweep
&lt;/h2&gt;

&lt;p&gt;A simple benchmark matrix could look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;input_tokens&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="m"&gt;256&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="m"&gt;2048&lt;/span&gt;

&lt;span class="na"&gt;output_tokens&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;128&lt;/span&gt;

&lt;span class="na"&gt;concurrency&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="m"&gt;4&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="m"&gt;8&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="m"&gt;16&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="m"&gt;32&lt;/span&gt;

&lt;span class="na"&gt;repeats&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;
&lt;span class="na"&gt;stream&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;temperature&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact values should match your real workload.&lt;/p&gt;

&lt;p&gt;If production requests usually have:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8k input tokens
+
1k output tokens
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;then a benchmark using:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;256 input tokens
+
64 output tokens
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;may tell you very little about production behavior.&lt;/p&gt;




&lt;h2&gt;
  
  
  Increase Concurrency Until Saturation
&lt;/h2&gt;

&lt;p&gt;Start at:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;concurrency = 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then increase gradually.&lt;/p&gt;

&lt;p&gt;For every level:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;warm up the server&lt;/li&gt;
&lt;li&gt;execute several repeated runs&lt;/li&gt;
&lt;li&gt;collect latency&lt;/li&gt;
&lt;li&gt;collect throughput&lt;/li&gt;
&lt;li&gt;collect errors&lt;/li&gt;
&lt;li&gt;collect queue metrics&lt;/li&gt;
&lt;li&gt;record GPU utilization&lt;/li&gt;
&lt;li&gt;record GPU memory usage&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Continue until you reach the first unacceptable condition.&lt;/p&gt;

&lt;p&gt;Examples include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OOM
Timeout
HTTP 503
Rejected request
p99 &amp;gt; SLO
GPU memory exhaustion
Queue growth without recovery
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That point is much more useful than the absolute maximum throughput.&lt;/p&gt;




&lt;h2&gt;
  
  
  Closed-Loop vs Open-Loop Testing
&lt;/h2&gt;

&lt;p&gt;There are two common ways to generate concurrent load.&lt;/p&gt;

&lt;p&gt;They measure different things.&lt;/p&gt;

&lt;h3&gt;
  
  
  Closed-Loop Load
&lt;/h3&gt;

&lt;p&gt;A closed-loop generator maintains a fixed number of active requests.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Concurrency = 16
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Whenever one request finishes, another starts.&lt;/p&gt;

&lt;p&gt;This answers questions such as:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;How does the server behave with 16 continuously active clients?&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h3&gt;
  
  
  Open-Loop Load
&lt;/h3&gt;

&lt;p&gt;An open-loop generator sends requests at a defined arrival rate.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;20 requests/second
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Requests continue arriving regardless of how quickly previous requests finish.&lt;/p&gt;

&lt;p&gt;This is useful for exposing queueing and overload behavior.&lt;/p&gt;

&lt;p&gt;If the server can process:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;15 requests/second
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;while receiving:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;20 requests/second
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;then approximately:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;5 requests/second
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;are being added to the queue.&lt;/p&gt;

&lt;p&gt;The queue may continue growing until:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;latency becomes unacceptable&lt;/li&gt;
&lt;li&gt;requests time out&lt;/li&gt;
&lt;li&gt;requests are rejected&lt;/li&gt;
&lt;li&gt;memory is exhausted&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not combine closed-loop and open-loop measurements into a single curve.&lt;/p&gt;

&lt;p&gt;They answer different questions.&lt;/p&gt;




&lt;h2&gt;
  
  
  Throughput Alone Is Not Enough
&lt;/h2&gt;

&lt;p&gt;Aggregate tokens per second describes server capacity.&lt;/p&gt;

&lt;p&gt;It does not fully describe the user experience.&lt;/p&gt;

&lt;p&gt;For every concurrency level, collect:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;requests per second&lt;/li&gt;
&lt;li&gt;output tokens per second&lt;/li&gt;
&lt;li&gt;TTFT&lt;/li&gt;
&lt;li&gt;ITL&lt;/li&gt;
&lt;li&gt;TPOT&lt;/li&gt;
&lt;li&gt;end-to-end latency&lt;/li&gt;
&lt;li&gt;successful-request rate&lt;/li&gt;
&lt;li&gt;queue length&lt;/li&gt;
&lt;li&gt;memory usage&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You need both throughput and latency.&lt;/p&gt;




&lt;h2&gt;
  
  
  TTFT: Time to First Token
&lt;/h2&gt;

&lt;p&gt;TTFT measures:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Request sent
      ↓
First actual generated token
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do not automatically treat the first streaming chunk as the first token.&lt;/p&gt;

&lt;p&gt;Some APIs may send metadata or other fields before actual generated text appears.&lt;/p&gt;

&lt;p&gt;Measure the first real generated output.&lt;/p&gt;

&lt;p&gt;TTFT strongly affects perceived responsiveness in chat applications.&lt;/p&gt;

&lt;p&gt;A user may tolerate a long generation if text begins appearing quickly.&lt;/p&gt;

&lt;p&gt;A long silent wait before the first token often feels much worse.&lt;/p&gt;




&lt;h2&gt;
  
  
  ITL: Inter-Token Latency
&lt;/h2&gt;

&lt;p&gt;ITL describes the delay between generated tokens.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Token 1
   ↓ 40 ms
Token 2
   ↓ 43 ms
Token 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This describes how smoothly the response streams after generation begins.&lt;/p&gt;

&lt;p&gt;If the client measures network chunks rather than actual tokens, call the measurement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;inter-chunk latency
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;instead.&lt;/p&gt;




&lt;h2&gt;
  
  
  TPOT: Time per Output Token
&lt;/h2&gt;

&lt;p&gt;TPOT can be represented conceptually as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;generation time after first token
---------------------------------
remaining output tokens
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It helps separate initial responsiveness from steady-state generation speed.&lt;/p&gt;




&lt;h2&gt;
  
  
  Measure End-to-End Latency
&lt;/h2&gt;

&lt;p&gt;End-to-end latency measures:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Request sent
      ↓
Queueing
      ↓
Prefill
      ↓
Generation
      ↓
Final response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is often the metric most directly connected to the total user wait time.&lt;/p&gt;




&lt;h2&gt;
  
  
  Always Measure Tail Latency
&lt;/h2&gt;

&lt;p&gt;Do not report only the average.&lt;/p&gt;

&lt;p&gt;At minimum, report:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;p50
p95
p99
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why?&lt;/p&gt;

&lt;p&gt;Because batching and queueing may affect a minority of requests much more severely than the median request.&lt;/p&gt;

&lt;p&gt;Suppose:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;p50 = 1.8 s
p95 = 4.2 s
p99 = 12.7 s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An average of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2.4 s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;would hide an important part of the user experience.&lt;/p&gt;

&lt;p&gt;For production inference APIs, p99 often matters much more than the mean.&lt;/p&gt;




&lt;h2&gt;
  
  
  Understand the Throughput-Latency Trade-Off
&lt;/h2&gt;

&lt;p&gt;Continuous batching can improve aggregate throughput.&lt;/p&gt;

&lt;p&gt;But that does not mean every request becomes faster.&lt;/p&gt;

&lt;p&gt;Consider:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Concurrency 4
Throughput: 3,000 tok/s
p99 TTFT: 400 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Concurrency 32
Throughput: 7,500 tok/s
p99 TTFT: 4.8 s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second configuration produces much more aggregate throughput.&lt;/p&gt;

&lt;p&gt;But it may violate the application's SLO.&lt;/p&gt;

&lt;p&gt;If your requirement is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;p99 TTFT &amp;lt; 2 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;then the higher-throughput operating point is not acceptable.&lt;/p&gt;

&lt;p&gt;The best benchmark result is therefore not necessarily the highest point on the throughput curve.&lt;/p&gt;

&lt;p&gt;It is the highest useful operating point &lt;strong&gt;inside the SLO&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  Measure Queueing Explicitly
&lt;/h2&gt;

&lt;p&gt;Queueing is one of the most important parts of concurrent inference.&lt;/p&gt;

&lt;p&gt;Imagine requests arriving faster than they can be processed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Arrival rate
     ↓
[Request]
[Request]
[Request]
[Request]
     ↓
Server
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the server cannot keep up:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Queue length ↑
TTFT ↑
p99 ↑
timeouts ↑
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is why a large or unlimited queue does not solve overload.&lt;/p&gt;

&lt;p&gt;It only changes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;rejected request
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;into:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;very slow request
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Backpressure and overload behavior should therefore be part of the benchmark.&lt;/p&gt;




&lt;h2&gt;
  
  
  Test Fairness Between Short and Long Requests
&lt;/h2&gt;

&lt;p&gt;A realistic workload rarely contains identical prompts.&lt;/p&gt;

&lt;p&gt;Create a mixed workload with:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;short inputs&lt;/li&gt;
&lt;li&gt;long inputs&lt;/li&gt;
&lt;li&gt;short outputs&lt;/li&gt;
&lt;li&gt;long outputs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then check whether short requests are excessively delayed by long ones.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Request A:
256 input tokens
64 output tokens

Request B:
8,000 input tokens
2,000 output tokens
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A scheduler should not allow a large request to make every small request unreasonably slow.&lt;/p&gt;

&lt;p&gt;Also observe whether large prefill operations create visible pauses for requests already generating tokens.&lt;/p&gt;




&lt;h2&gt;
  
  
  Store Raw Benchmark Data
&lt;/h2&gt;

&lt;p&gt;A professional benchmark should produce reproducible artifacts.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;benchmark/
├── requests.jsonl
├── run.json
├── results.csv
├── server.log
└── gpu.csv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A minimal result schema could look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;server,run,concurrency,input_tokens,output_tokens,ttft_ms,itl_ms,e2e_ms,status,error
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You may also want:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;timestamp,gpu_utilization,gpu_memory_used,queue_length
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Screenshots alone are not sufficient.&lt;/p&gt;

&lt;p&gt;Raw data allows you to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;recalculate percentiles&lt;/li&gt;
&lt;li&gt;change aggregation windows&lt;/li&gt;
&lt;li&gt;inspect outliers&lt;/li&gt;
&lt;li&gt;compare future server versions&lt;/li&gt;
&lt;li&gt;reproduce the analysis&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Graph Throughput and Latency Together
&lt;/h2&gt;

&lt;p&gt;A useful benchmark graph uses concurrency on the X-axis.&lt;/p&gt;

&lt;p&gt;Then plot metrics such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Requests/s
Output tokens/s
TTFT p50
TTFT p95
TTFT p99
End-to-end p99
Queue length
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The important shape often looks like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Throughput
    /
   /
  /
 /_____
       \
        saturation

Latency
        /
       /
______/ 
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At low concurrency:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;throughput increases&lt;/li&gt;
&lt;li&gt;latency remains stable&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Near saturation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;throughput growth slows&lt;/li&gt;
&lt;li&gt;queueing increases&lt;/li&gt;
&lt;li&gt;p99 rises rapidly&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That knee in the curve is often more important than the absolute maximum.&lt;/p&gt;




&lt;h2&gt;
  
  
  Compare API Compatibility Before Migrating
&lt;/h2&gt;

&lt;p&gt;Both Ollama and vLLM may expose OpenAI-compatible interfaces.&lt;/p&gt;

&lt;p&gt;But:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OpenAI-compatible
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;does not mean:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;behaviorally identical
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Verify the features your application actually uses.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Chat Completions&lt;/li&gt;
&lt;li&gt;streaming&lt;/li&gt;
&lt;li&gt;structured output&lt;/li&gt;
&lt;li&gt;embeddings&lt;/li&gt;
&lt;li&gt;tool calling&lt;/li&gt;
&lt;li&gt;generation parameters&lt;/li&gt;
&lt;li&gt;stop sequences&lt;/li&gt;
&lt;li&gt;error responses&lt;/li&gt;
&lt;li&gt;token limits&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not discover incompatibilities after production traffic has already been switched.&lt;/p&gt;




&lt;h2&gt;
  
  
  Production Readiness Is More Than the Inference Engine
&lt;/h2&gt;

&lt;p&gt;A working HTTP endpoint is not automatically a production-ready inference service.&lt;/p&gt;

&lt;p&gt;The surrounding platform may need to provide:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TLS
Authentication
Rate limiting
Logging
Monitoring
Health checks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A health check should also distinguish between:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Server process is running
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Model is loaded and ready to serve
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A listening TCP port does not guarantee that the next request will meet the latency SLO.&lt;/p&gt;




&lt;h2&gt;
  
  
  Compare Observability
&lt;/h2&gt;

&lt;p&gt;For concurrent production workloads, observability becomes part of the product decision.&lt;/p&gt;

&lt;p&gt;Useful vLLM signals may include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;queue state&lt;/li&gt;
&lt;li&gt;KV-cache utilization&lt;/li&gt;
&lt;li&gt;TTFT&lt;/li&gt;
&lt;li&gt;request latency&lt;/li&gt;
&lt;li&gt;memory pressure&lt;/li&gt;
&lt;li&gt;preemption&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Ollama also exposes timing information around:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;model loading&lt;/li&gt;
&lt;li&gt;prompt evaluation&lt;/li&gt;
&lt;li&gt;generation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Regardless of the server, collect the same client-side metrics.&lt;/p&gt;

&lt;p&gt;Client-side instrumentation provides the most direct view of what users actually experience.&lt;/p&gt;




&lt;h2&gt;
  
  
  Test Upgrades Before Production
&lt;/h2&gt;

&lt;p&gt;Inference-server updates can change:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;memory consumption&lt;/li&gt;
&lt;li&gt;scheduler behavior&lt;/li&gt;
&lt;li&gt;inference backends&lt;/li&gt;
&lt;li&gt;configuration defaults&lt;/li&gt;
&lt;li&gt;chat templates&lt;/li&gt;
&lt;li&gt;model compatibility&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Before upgrading production, replay a realistic workload against the new version.&lt;/p&gt;

&lt;p&gt;Keep the previous:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Container image or package
Configuration
Model hash
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;available for rollback.&lt;/p&gt;

&lt;p&gt;A benchmark that is valid for one version should not automatically be assumed valid for the next.&lt;/p&gt;




&lt;h2&gt;
  
  
  Use a Shared API Layer to Keep Migration Easy
&lt;/h2&gt;

&lt;p&gt;If possible, avoid coupling application logic directly to one inference engine.&lt;/p&gt;

&lt;p&gt;Keep:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;model names&lt;/li&gt;
&lt;li&gt;generation parameters&lt;/li&gt;
&lt;li&gt;timeout logic&lt;/li&gt;
&lt;li&gt;error handling&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;behind an internal adapter.&lt;/p&gt;

&lt;p&gt;Then make the server endpoint configurable.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application
     ↓
Internal inference adapter
     ↓
Ollama OR vLLM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This makes it much easier to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;benchmark alternatives&lt;/li&gt;
&lt;li&gt;canary a new server&lt;/li&gt;
&lt;li&gt;roll back&lt;/li&gt;
&lt;li&gt;migrate later&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Canary the Migration
&lt;/h2&gt;

&lt;p&gt;Do not send 100% of production traffic to the new server immediately.&lt;/p&gt;

&lt;p&gt;Start with a small percentage.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;5% new server
95% old server
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Measure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;latency&lt;/li&gt;
&lt;li&gt;errors&lt;/li&gt;
&lt;li&gt;throughput&lt;/li&gt;
&lt;li&gt;queue growth&lt;/li&gt;
&lt;li&gt;memory usage&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then increase gradually if the results remain inside the SLO.&lt;/p&gt;

&lt;p&gt;Keep the old server available for rapid rollback.&lt;/p&gt;




&lt;h2&gt;
  
  
  A Practical Benchmark Workflow
&lt;/h2&gt;

&lt;p&gt;Here is a reusable process.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Define the Real Workload
&lt;/h3&gt;

&lt;p&gt;Record:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Input length
Output length
Average concurrency
Peak concurrency
Request rate
Streaming ratio
SLO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 2: Normalize the Models
&lt;/h3&gt;

&lt;p&gt;Match:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Weights
Precision
Quantization
Tokenizer
Chat template
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 3: Match Generation Settings
&lt;/h3&gt;

&lt;p&gt;Use identical:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;max_tokens
temperature
top_p
stop
stream
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 4: Record the Environment
&lt;/h3&gt;

&lt;p&gt;Save:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Server version
Model hash
GPU
VRAM
CPU
RAM
OS
Driver
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 5: Warm Up the Server
&lt;/h3&gt;

&lt;p&gt;Separate cold-start measurements from steady-state results.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 6: Run the Concurrency Sweep
&lt;/h3&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1
2
4
8
16
32
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 7: Measure Latency and Throughput
&lt;/h3&gt;

&lt;p&gt;Collect:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Requests/s
Tokens/s
TTFT
ITL
TPOT
E2E latency
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 8: Measure Tail Latency
&lt;/h3&gt;

&lt;p&gt;Calculate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;p50
p95
p99
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 9: Watch Queue and GPU State
&lt;/h3&gt;

&lt;p&gt;Collect:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Queue length
GPU utilization
GPU memory
KV-cache
OOM events
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 10: Stop at the SLO Boundary
&lt;/h3&gt;

&lt;p&gt;The benchmark should stop being considered "better" once:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;p99 &amp;gt; SLO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;even if tokens per second continue increasing.&lt;/p&gt;




&lt;h2&gt;
  
  
  Starting Recommendation by Scenario
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Single User
&lt;/h3&gt;

&lt;p&gt;Start with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Ollama
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Confirm:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the model is supported&lt;/li&gt;
&lt;li&gt;cold start is acceptable&lt;/li&gt;
&lt;li&gt;memory consumption is acceptable&lt;/li&gt;
&lt;li&gt;latency is acceptable&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The simplicity may be worth more than extra concurrent throughput.&lt;/p&gt;




&lt;h3&gt;
  
  
  Small Internal API
&lt;/h3&gt;

&lt;p&gt;Start with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Ollama or vLLM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Benchmark:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;request bursts&lt;/li&gt;
&lt;li&gt;moderate concurrency&lt;/li&gt;
&lt;li&gt;queue behavior&lt;/li&gt;
&lt;li&gt;p95/p99&lt;/li&gt;
&lt;li&gt;overload handling&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Either system may be the better fit.&lt;/p&gt;




&lt;h3&gt;
  
  
  High-Concurrency GPU Service
&lt;/h3&gt;

&lt;p&gt;Start with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;vLLM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then confirm that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;throughput improves with concurrency&lt;/li&gt;
&lt;li&gt;p99 remains inside the SLO&lt;/li&gt;
&lt;li&gt;TTFT remains acceptable&lt;/li&gt;
&lt;li&gt;KV-cache remains healthy&lt;/li&gt;
&lt;li&gt;queue growth remains controlled&lt;/li&gt;
&lt;li&gt;OOM does not occur&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not assume the answer before measuring it.&lt;/p&gt;




&lt;h2&gt;
  
  
  Practical Checklist
&lt;/h2&gt;

&lt;p&gt;Before declaring one server faster than the other:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Use comparable model weights&lt;/li&gt;
&lt;li&gt;[ ] Match precision and quantization&lt;/li&gt;
&lt;li&gt;[ ] Match tokenizer behavior&lt;/li&gt;
&lt;li&gt;[ ] Match chat templates&lt;/li&gt;
&lt;li&gt;[ ] Match generation parameters&lt;/li&gt;
&lt;li&gt;[ ] Record software versions&lt;/li&gt;
&lt;li&gt;[ ] Record GPU and VRAM&lt;/li&gt;
&lt;li&gt;[ ] Separate cold and warm tests&lt;/li&gt;
&lt;li&gt;[ ] Use realistic input lengths&lt;/li&gt;
&lt;li&gt;[ ] Use realistic output lengths&lt;/li&gt;
&lt;li&gt;[ ] Sweep concurrency&lt;/li&gt;
&lt;li&gt;[ ] Test open-loop and closed-loop load separately&lt;/li&gt;
&lt;li&gt;[ ] Measure requests per second&lt;/li&gt;
&lt;li&gt;[ ] Measure output tokens per second&lt;/li&gt;
&lt;li&gt;[ ] Measure TTFT&lt;/li&gt;
&lt;li&gt;[ ] Measure ITL or TPOT&lt;/li&gt;
&lt;li&gt;[ ] Measure end-to-end latency&lt;/li&gt;
&lt;li&gt;[ ] Calculate p50/p95/p99&lt;/li&gt;
&lt;li&gt;[ ] Monitor queue length&lt;/li&gt;
&lt;li&gt;[ ] Monitor GPU utilization&lt;/li&gt;
&lt;li&gt;[ ] Monitor GPU memory&lt;/li&gt;
&lt;li&gt;[ ] Record OOM and rejected requests&lt;/li&gt;
&lt;li&gt;[ ] Preserve raw benchmark data&lt;/li&gt;
&lt;li&gt;[ ] Evaluate results against the SLO&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Final Takeaway
&lt;/h2&gt;

&lt;p&gt;Ollama and vLLM solve overlapping problems, but they optimize for different priorities.&lt;/p&gt;

&lt;p&gt;Ollama is attractive when you value:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;fast setup&lt;/li&gt;
&lt;li&gt;simple model management&lt;/li&gt;
&lt;li&gt;a straightforward local lifecycle&lt;/li&gt;
&lt;li&gt;moderate concurrency&lt;/li&gt;
&lt;li&gt;lower operational complexity&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;vLLM becomes more attractive when you need:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;continuous batching&lt;/li&gt;
&lt;li&gt;higher concurrent throughput&lt;/li&gt;
&lt;li&gt;efficient KV-cache management&lt;/li&gt;
&lt;li&gt;high GPU utilization&lt;/li&gt;
&lt;li&gt;larger request queues&lt;/li&gt;
&lt;li&gt;detailed production observability&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The correct comparison is not:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which server produces the most tokens per second?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which server delivers the throughput you need while keeping TTFT, p99 latency, queueing, errors, memory usage, and operational complexity inside acceptable limits?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If vLLM produces higher aggregate throughput but pushes p99 beyond your SLO, that is not a win.&lt;/p&gt;

&lt;p&gt;If Ollama already handles your real traffic comfortably, migrating to a more complex serving stack may provide little practical benefit.&lt;/p&gt;

&lt;p&gt;Benchmark the workload you actually have.&lt;/p&gt;

&lt;p&gt;Find the saturation point.&lt;/p&gt;

&lt;p&gt;Measure the queue.&lt;/p&gt;

&lt;p&gt;Measure the tails.&lt;/p&gt;

&lt;p&gt;Then choose the simplest system that satisfies the SLO.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>devops</category>
      <category>performance</category>
    </item>
    <item>
      <title>CPU Steal Time Explained: What %st Really Means on a Virtual Machine</title>
      <dc:creator>Aeza</dc:creator>
      <pubDate>Thu, 03 Sep 2026 14:11:28 +0000</pubDate>
      <link>https://dev.to/aeza__net/cpu-steal-time-explained-what-st-really-means-on-a-virtual-machine-109h</link>
      <guid>https://dev.to/aeza__net/cpu-steal-time-explained-what-st-really-means-on-a-virtual-machine-109h</guid>
      <description>&lt;p&gt;CPU steal time is one of those Linux metrics that looks simple until you actually need to diagnose a performance problem.&lt;/p&gt;

&lt;p&gt;You open &lt;code&gt;top&lt;/code&gt;, notice that &lt;code&gt;%st&lt;/code&gt; is above zero, and the obvious conclusion seems to be:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The host is overloaded.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Or perhaps:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Another VM on the server is stealing my CPU.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Both explanations are possible.&lt;/p&gt;

&lt;p&gt;Neither can be proven from &lt;code&gt;%st&lt;/code&gt; alone.&lt;/p&gt;

&lt;p&gt;A useful diagnosis requires at least three things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the steal-time measurement itself&lt;/li&gt;
&lt;li&gt;CPU pressure inside the VM&lt;/li&gt;
&lt;li&gt;measurable impact on the application&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In practice, that means correlating &lt;code&gt;%st&lt;/code&gt; with metrics such as the run queue, p95/p99 latency, throughput, per-vCPU utilization, and cgroup throttling.&lt;/p&gt;

&lt;p&gt;In this article, we'll build a practical workflow for doing exactly that.&lt;/p&gt;

&lt;p&gt;By the end, you'll know:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;what &lt;code&gt;%st&lt;/code&gt; actually measures&lt;/li&gt;
&lt;li&gt;how to collect steal-time data correctly&lt;/li&gt;
&lt;li&gt;when high steal time is worth investigating&lt;/li&gt;
&lt;li&gt;how to separate host contention from guest CPU saturation&lt;/li&gt;
&lt;li&gt;what evidence to collect before contacting your hosting provider&lt;/li&gt;
&lt;li&gt;how to monitor steal time continuously&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  The Short Answer
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;%st&lt;/code&gt; represents time during which a virtual CPU was ready to run but did not receive physical CPU time from the host.&lt;/p&gt;

&lt;p&gt;But a high value only becomes operationally meaningful when it is:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;abnormal for that specific VM&lt;/li&gt;
&lt;li&gt;sustained across multiple measurement intervals&lt;/li&gt;
&lt;li&gt;correlated with CPU pressure or application degradation&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A single spike is not enough.&lt;/p&gt;

&lt;p&gt;A much stronger signal looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Elevated %st
+
Growing CPU run queue
+
Increasing p99 latency
+
Falling throughput
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That combination is worth investigating.&lt;/p&gt;




&lt;h2&gt;
  
  
  What Does &lt;code&gt;%st&lt;/code&gt; Mean in &lt;code&gt;top&lt;/code&gt; and &lt;code&gt;vmstat&lt;/code&gt;?
&lt;/h2&gt;

&lt;p&gt;In a virtual machine, &lt;code&gt;%st&lt;/code&gt; — or steal time — represents the percentage of a measurement interval during which a vCPU was ready to execute but did not receive physical CPU time.&lt;/p&gt;

&lt;p&gt;You may see it in &lt;code&gt;top&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%Cpu(s): 12.7 us, 6.4 sy, 0.0 ni, 72.2 id,
          3.1 wa, 0.0 hi, 0.0 si, 5.6 st
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In this example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;5.6 st
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;means that approximately 5.6% of the measured CPU interval was accounted as steal time.&lt;/p&gt;

&lt;p&gt;The important part is understanding &lt;em&gt;why&lt;/em&gt; this can happen.&lt;/p&gt;

&lt;p&gt;A VM does not control the physical CPU directly.&lt;/p&gt;

&lt;p&gt;There is another scheduling layer underneath it.&lt;/p&gt;




&lt;h2&gt;
  
  
  A Virtual Machine Has Two CPU Schedulers
&lt;/h2&gt;

&lt;p&gt;To understand steal time, separate the guest scheduler from the host scheduler.&lt;/p&gt;

&lt;h3&gt;
  
  
  The guest scheduler
&lt;/h3&gt;

&lt;p&gt;The operating system inside the VM decides:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which process should run on this vCPU?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;nginx
postgres
python
node
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;may all compete for CPU time inside the guest.&lt;/p&gt;

&lt;h3&gt;
  
  
  The host scheduler
&lt;/h3&gt;

&lt;p&gt;The physical host or hypervisor decides something different:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;When does this vCPU get access to an actual physical CPU?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That gives us two scheduling layers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application process
        ↓
Guest OS scheduler
        ↓
       vCPU
        ↓
Host / hypervisor scheduler
        ↓
Physical CPU
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Suppose the guest has a runnable process.&lt;/p&gt;

&lt;p&gt;The guest scheduler wants to execute it.&lt;/p&gt;

&lt;p&gt;The vCPU is therefore ready.&lt;/p&gt;

&lt;p&gt;But the host scheduler does not immediately schedule that vCPU onto a physical core.&lt;/p&gt;

&lt;p&gt;The guest waits.&lt;/p&gt;

&lt;p&gt;That waiting time can be recorded as steal time.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Guest has runnable work
        ↓
vCPU is ready
        ↓
Physical CPU is not assigned
        ↓
Guest waits
        ↓
Time is recorded as steal
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is why &lt;code&gt;%st&lt;/code&gt; applies to the vCPU as a whole rather than to one particular process.&lt;/p&gt;




&lt;h2&gt;
  
  
  Do Not Confuse &lt;code&gt;%st&lt;/code&gt; with Other CPU Metrics
&lt;/h2&gt;

&lt;p&gt;Several CPU metrics can indicate performance pressure, but they describe different things.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;%usr&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Time spent executing user-space code.&lt;/p&gt;

&lt;p&gt;Examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;application logic&lt;/li&gt;
&lt;li&gt;JavaScript&lt;/li&gt;
&lt;li&gt;Python&lt;/li&gt;
&lt;li&gt;database execution&lt;/li&gt;
&lt;li&gt;compression&lt;/li&gt;
&lt;li&gt;model inference&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;%sys&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Time spent executing kernel code.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;%idle&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;CPU time during which the guest had no runnable work.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;%iowait&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Time associated with waiting while I/O is pending.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;%st&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Time during which a runnable virtual CPU did not receive physical CPU time.&lt;/p&gt;

&lt;p&gt;A useful mental model is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%usr / %sys → guest is actively using CPU

%iowait     → work is waiting around I/O

%st         → guest wanted CPU but the host did not provide it
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;%iowait&lt;/code&gt; is therefore &lt;strong&gt;not&lt;/strong&gt; another form of steal time.&lt;/p&gt;




&lt;h2&gt;
  
  
  CPU Throttling Is Also Not the Same as Steal Time
&lt;/h2&gt;

&lt;p&gt;A process can be CPU-limited inside the VM even when &lt;code&gt;%st&lt;/code&gt; is zero.&lt;/p&gt;

&lt;p&gt;One common cause is a cgroup CPU quota.&lt;/p&gt;

&lt;p&gt;With cgroup v2, inspect the configured quota:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/cpu.max
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then inspect CPU statistics:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/cpu.stat
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Useful counters can include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;nr_throttled
throttled_usec
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If these counters increase while the workload is running, the process or cgroup may be hitting a CPU quota.&lt;/p&gt;

&lt;p&gt;That means the CPU restriction originates &lt;strong&gt;inside the guest environment&lt;/strong&gt;, not necessarily from hypervisor scheduling.&lt;/p&gt;

&lt;p&gt;So this situation is possible:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st = 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;while application latency still increases because the process is being throttled.&lt;/p&gt;

&lt;p&gt;This distinction matters when trying to determine whether the VM itself is overloaded or whether the underlying platform is contributing to the problem.&lt;/p&gt;




&lt;h2&gt;
  
  
  A Zero &lt;code&gt;%st&lt;/code&gt; Does Not Prove the Host Is Healthy
&lt;/h2&gt;

&lt;p&gt;Another common mistake is assuming:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st = 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;means:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;No host-side scheduling problems exist
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That conclusion is too strong.&lt;/p&gt;

&lt;p&gt;A zero value only tells you that the guest kernel did not account for steal time during that particular interval.&lt;/p&gt;

&lt;p&gt;The availability of steal-time accounting depends on factors such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;hypervisor&lt;/li&gt;
&lt;li&gt;guest kernel&lt;/li&gt;
&lt;li&gt;architecture&lt;/li&gt;
&lt;li&gt;platform configuration&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;KVM can expose steal time on supported systems, but the guest still needs access to the relevant accounting information.&lt;/p&gt;

&lt;p&gt;Other forms of host-side delay may also exist without appearing as &lt;code&gt;%st&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;So think of zero as:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;No steal time was recorded here.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Not:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The physical host is definitely healthy.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Measure Steal Time over Intervals
&lt;/h2&gt;

&lt;p&gt;One of the easiest ways to misread steal time is to use the wrong measurement window.&lt;/p&gt;

&lt;p&gt;For example, the first row printed by &lt;code&gt;vmstat&lt;/code&gt; can represent averages since system boot unless you suppress it.&lt;/p&gt;

&lt;p&gt;A since-boot average is nearly useless when investigating a short incident.&lt;/p&gt;

&lt;p&gt;Imagine that your VM has been running for 30 days.&lt;/p&gt;

&lt;p&gt;A two-minute scheduling problem may almost disappear inside the long-term average.&lt;/p&gt;

&lt;p&gt;For incident analysis, collect interval-based measurements instead.&lt;/p&gt;




&lt;h2&gt;
  
  
  Collect One Minute of &lt;code&gt;vmstat&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;A useful starting point is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C vmstat &lt;span class="nt"&gt;-y&lt;/span&gt; &lt;span class="nt"&gt;-t&lt;/span&gt; 1 60 | &lt;span class="nb"&gt;tee &lt;/span&gt;vmstat.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This gives you:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 sample per second
×
60 samples
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Using &lt;code&gt;-y&lt;/code&gt; skips the misleading initial since-boot row.&lt;/p&gt;

&lt;p&gt;The output allows you to observe steal time together with metrics such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;r&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;us&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;sy&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;id&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;wa&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;st&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;during exactly the same intervals.&lt;/p&gt;




&lt;h2&gt;
  
  
  Collect Per-vCPU Data with &lt;code&gt;mpstat&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;A system-wide CPU average can hide an important detail.&lt;/p&gt;

&lt;p&gt;Suppose a four-vCPU machine looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CPU 0 → heavily delayed
CPU 1 → mostly idle
CPU 2 → mostly idle
CPU 3 → mostly idle
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The average across all CPUs may look relatively normal.&lt;/p&gt;

&lt;p&gt;To inspect individual virtual CPUs, use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C mpstat &lt;span class="nt"&gt;-P&lt;/span&gt; ALL 1 60 | &lt;span class="nb"&gt;tee &lt;/span&gt;mpstat.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This lets you compare each vCPU separately.&lt;/p&gt;

&lt;p&gt;That becomes especially important for workloads that rely heavily on a small number of threads.&lt;/p&gt;




&lt;h2&gt;
  
  
  What Does the &lt;code&gt;r&lt;/code&gt; Column in &lt;code&gt;vmstat&lt;/code&gt; Mean?
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;r&lt;/code&gt; field represents runnable tasks.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;r =
tasks currently executing
+
tasks ready and waiting for CPU
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is not an exact queue size for each individual CPU core.&lt;/p&gt;

&lt;p&gt;But it is useful as a CPU-pressure signal.&lt;/p&gt;

&lt;p&gt;Consider these two situations.&lt;/p&gt;

&lt;h3&gt;
  
  
  Situation A
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st: brief spike
r: normal
p99: normal
throughput: normal
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There may be nothing operationally important happening.&lt;/p&gt;

&lt;h3&gt;
  
  
  Situation B
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st: elevated
r: increasing
p99: increasing
throughput: falling
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the evidence is much stronger.&lt;/p&gt;

&lt;p&gt;The second case tells us that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;the VM is waiting for CPU&lt;/li&gt;
&lt;li&gt;runnable work is accumulating&lt;/li&gt;
&lt;li&gt;users are experiencing measurable degradation&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That is the kind of correlation worth investigating.&lt;/p&gt;




&lt;h2&gt;
  
  
  Record the Environment with Every Incident
&lt;/h2&gt;

&lt;p&gt;Raw performance numbers are much less useful without environmental context.&lt;/p&gt;

&lt;p&gt;At minimum, collect the timestamp:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; &lt;span class="nt"&gt;--iso-8601&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Kernel version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;uname&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Number of processors visible to the guest:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;nproc&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Virtualization type:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemd-detect-virt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And hypervisor information where available:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C lscpu | &lt;span class="nb"&gt;sed&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s1"&gt;'/Hypervisor vendor/p;/Virtualization type/p'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Store at least:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;UTC timestamp
Kernel version
Number of vCPUs
Virtualization type
VM plan
CPU limits
Region
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This matters because two results such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st = 8%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;may describe completely different environments.&lt;/p&gt;

&lt;p&gt;An 8% reading on a one-vCPU burstable instance is not automatically comparable to 8% on an eight-vCPU VM with a different CPU policy.&lt;/p&gt;




&lt;h2&gt;
  
  
  Establish a Baseline Before Calling &lt;code&gt;%st&lt;/code&gt; High
&lt;/h2&gt;

&lt;p&gt;There is no universal steal-time threshold.&lt;/p&gt;

&lt;p&gt;You will sometimes see rules such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st &amp;gt; 5% = bad
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st &amp;gt; 10% = overloaded host
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These rules are convenient.&lt;/p&gt;

&lt;p&gt;They are also too simplistic.&lt;/p&gt;

&lt;p&gt;A better question is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Is this value abnormal for this VM under a comparable workload?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Build a baseline using measurements collected during:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;low-load periods&lt;/li&gt;
&lt;li&gt;normal-load periods&lt;/li&gt;
&lt;li&gt;comparable hours&lt;/li&gt;
&lt;li&gt;comparable weekdays&lt;/li&gt;
&lt;li&gt;similar request volumes&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then calculate values such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;median
p95
p99
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;for the steal-time series.&lt;/p&gt;

&lt;p&gt;The goal is to understand the normal distribution for the machine.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why the Maximum Is Often a Bad Metric
&lt;/h2&gt;

&lt;p&gt;Suppose you observe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Maximum %st this month: 35%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That sounds alarming.&lt;/p&gt;

&lt;p&gt;But what if the 35% value lasted for one second and nothing happened to the application?&lt;/p&gt;

&lt;p&gt;Now compare that with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st: 7–10%
Duration: 12 minutes
p99 latency: +70%
Throughput: -20%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second event may be much more important even though its maximum value is lower.&lt;/p&gt;

&lt;p&gt;For performance incidents, persistence and application impact often matter more than the highest isolated value.&lt;/p&gt;




&lt;h2&gt;
  
  
  Rebuild the Baseline After Major VM Changes
&lt;/h2&gt;

&lt;p&gt;The old baseline may stop being meaningful after changing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;number of vCPUs&lt;/li&gt;
&lt;li&gt;kernel version&lt;/li&gt;
&lt;li&gt;VM class&lt;/li&gt;
&lt;li&gt;service plan&lt;/li&gt;
&lt;li&gt;CPU limits&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, moving from:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2 vCPU
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 vCPU
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;changes thread distribution and the way aggregate CPU statistics should be interpreted.&lt;/p&gt;

&lt;p&gt;Treat significant configuration changes as the beginning of a new baseline period.&lt;/p&gt;




&lt;h2&gt;
  
  
  When Should You Investigate High Steal Time?
&lt;/h2&gt;

&lt;p&gt;A useful rule is to investigate when all three conditions begin to appear.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. &lt;code&gt;%st&lt;/code&gt; is above normal
&lt;/h3&gt;

&lt;p&gt;Not simply above an arbitrary internet threshold.&lt;/p&gt;

&lt;p&gt;It should be above the historical level for that VM.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. The condition persists
&lt;/h3&gt;

&lt;p&gt;Several consecutive elevated intervals are more important than a single isolated sample.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Something measurable becomes worse
&lt;/h3&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU run queue increases&lt;/li&gt;
&lt;li&gt;p95 latency increases&lt;/li&gt;
&lt;li&gt;p99 latency increases&lt;/li&gt;
&lt;li&gt;throughput decreases&lt;/li&gt;
&lt;li&gt;errors increase&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A weak signal might look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st: 12%
Duration: 1 second
p99: unchanged
Throughput: unchanged
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A stronger signal might look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st: elevated for 8 minutes
r: increased
p99: +80%
Throughput: -25%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second case is much more actionable.&lt;/p&gt;




&lt;h2&gt;
  
  
  Separate Host Contention from Guest CPU Saturation
&lt;/h2&gt;

&lt;p&gt;Steal time does not prevent the guest from also being CPU-bound.&lt;/p&gt;

&lt;p&gt;Both can happen at the same time.&lt;/p&gt;

&lt;p&gt;Start by examining:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%usr
%sys
%idle
r
%st
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Suppose you observe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;High %usr
High %sys
Low %idle
r &amp;gt; number of vCPUs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The guest itself may already be CPU saturated.&lt;/p&gt;

&lt;p&gt;That does not mean &lt;code&gt;%st&lt;/code&gt; is irrelevant.&lt;/p&gt;

&lt;p&gt;It means you may have multiple sources of delay.&lt;/p&gt;




&lt;h2&gt;
  
  
  Inspect Individual vCPUs
&lt;/h2&gt;

&lt;p&gt;Run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C mpstat &lt;span class="nt"&gt;-P&lt;/span&gt; ALL 1 60
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This can expose per-vCPU imbalance that disappears from the average.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CPU 0 → 100% busy
CPU 1 → 20%
CPU 2 → 15%
CPU 3 → 10%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A single CPU-bound thread may saturate one virtual CPU while the rest remain mostly free.&lt;/p&gt;




&lt;h2&gt;
  
  
  Inspect Application Threads
&lt;/h2&gt;

&lt;p&gt;Use &lt;code&gt;pidstat&lt;/code&gt; to inspect thread-level CPU usage:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C pidstat &lt;span class="nt"&gt;-u&lt;/span&gt; &lt;span class="nt"&gt;-t&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; PID 1 60
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This helps answer questions such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Is one thread consuming nearly all available CPU?&lt;/li&gt;
&lt;li&gt;Is CPU usage distributed evenly?&lt;/li&gt;
&lt;li&gt;Is the application itself creating the bottleneck?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Only after checking these guest-side causes does it become safer to focus on the host.&lt;/p&gt;




&lt;h2&gt;
  
  
  Do Not Ignore I/O, Locks, or External Dependencies
&lt;/h2&gt;

&lt;p&gt;High latency does not automatically mean CPU contention.&lt;/p&gt;

&lt;p&gt;Other possibilities include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;storage latency&lt;/li&gt;
&lt;li&gt;database locking&lt;/li&gt;
&lt;li&gt;application locks&lt;/li&gt;
&lt;li&gt;garbage collection&lt;/li&gt;
&lt;li&gt;network latency&lt;/li&gt;
&lt;li&gt;external APIs&lt;/li&gt;
&lt;li&gt;DNS&lt;/li&gt;
&lt;li&gt;remote databases&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So remember:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;High latency ≠ automatically CPU contention
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;High %st ≠ automatically a noisy neighbor
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A useful diagnosis should eliminate alternative explanations before assigning a cause.&lt;/p&gt;




&lt;h2&gt;
  
  
  Correlate &lt;code&gt;%st&lt;/code&gt; with Application Metrics
&lt;/h2&gt;

&lt;p&gt;The most useful analysis begins when system metrics and application metrics share the same timeline.&lt;/p&gt;

&lt;p&gt;Use UTC for everything.&lt;/p&gt;

&lt;p&gt;For each interval, compare:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st
r
request rate
throughput
error rate
p50 latency
p95 latency
p99 latency
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This allows you to answer a much better question than:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Was steal time high?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You can ask:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Did application performance become worse during exactly the same period?&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Do Not Compare Different Aggregation Windows
&lt;/h2&gt;

&lt;p&gt;Imagine this comparison:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st → one-second samples

p99 → five-minute window
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A CPU scheduling event lasting 30 seconds may disappear almost completely inside a five-minute application aggregate.&lt;/p&gt;

&lt;p&gt;The metrics then appear unrelated even when they describe the same incident.&lt;/p&gt;

&lt;p&gt;Try to align:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;timestamps&lt;/li&gt;
&lt;li&gt;collection intervals&lt;/li&gt;
&lt;li&gt;aggregation windows&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;as closely as possible.&lt;/p&gt;




&lt;h2&gt;
  
  
  A "Noisy Neighbor" Is a Hypothesis, Not a Measurement
&lt;/h2&gt;

&lt;p&gt;One of the most common explanations for steal time is another VM consuming physical CPU resources on the same host.&lt;/p&gt;

&lt;p&gt;That is the classic "noisy neighbor" scenario.&lt;/p&gt;

&lt;p&gt;It may be correct.&lt;/p&gt;

&lt;p&gt;But guest metrics alone cannot tell you:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;which VM caused the delay&lt;/li&gt;
&lt;li&gt;whether another tenant caused it&lt;/li&gt;
&lt;li&gt;whether the host itself was busy&lt;/li&gt;
&lt;li&gt;whether a scheduler policy was involved&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So instead of saying:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A noisy neighbor caused our outage.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A technically defensible statement would be:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;During this interval, the VM experienced increased CPU steal time while the run queue and application latency also increased.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That statement describes what you actually measured.&lt;/p&gt;

&lt;p&gt;Host telemetry is needed for stronger attribution.&lt;/p&gt;




&lt;h2&gt;
  
  
  When Does the Host-Contention Hypothesis Become Stronger?
&lt;/h2&gt;

&lt;p&gt;The hypothesis becomes more convincing when degradation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;repeats under similar traffic&lt;/li&gt;
&lt;li&gt;coincides with elevated &lt;code&gt;%st&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;is not explained by a deployment&lt;/li&gt;
&lt;li&gt;is not explained by garbage collection&lt;/li&gt;
&lt;li&gt;is not explained by I/O&lt;/li&gt;
&lt;li&gt;is not explained by cgroup throttling&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Host-side scheduling metrics provide the strongest confirmation.&lt;/p&gt;




&lt;h2&gt;
  
  
  Store Raw Data, Not Just Screenshots
&lt;/h2&gt;

&lt;p&gt;A useful incident dataset might include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;incident/
├── metadata.txt
├── vmstat.txt
├── mpstat.txt
├── application.csv
└── README.md
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A minimal application/system CSV could use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;timestamp_utc,st,r,usr,sys,p95_ms,p99_ms,rps,error_rate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Raw data lets you:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;recalculate percentiles&lt;/li&gt;
&lt;li&gt;change aggregation windows&lt;/li&gt;
&lt;li&gt;inspect individual events&lt;/li&gt;
&lt;li&gt;compare multiple incidents&lt;/li&gt;
&lt;li&gt;reproduce the analysis later&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A screenshot cannot do that.&lt;/p&gt;

&lt;p&gt;Once metrics have been compressed into an image, much of the original information is gone.&lt;/p&gt;




&lt;h2&gt;
  
  
  Test the Hypothesis with Controlled CPU Load
&lt;/h2&gt;

&lt;p&gt;A controlled test can help you understand how the VM behaves under a known CPU workload.&lt;/p&gt;

&lt;p&gt;First record the &lt;code&gt;stress-ng&lt;/code&gt; version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;stress-ng &lt;span class="nt"&gt;--version&lt;/span&gt; | &lt;span class="nb"&gt;tee &lt;/span&gt;stress-ng-version.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then run a CPU-bound workload pinned to one vCPU:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;taskset &lt;span class="nt"&gt;-c&lt;/span&gt; 0 stress-ng &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--cpu&lt;/span&gt; 1 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--cpu-method&lt;/span&gt; matrixprod &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--timeout&lt;/span&gt; 120s &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--metrics-brief&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  2&amp;gt;&amp;amp;1 | &lt;span class="nb"&gt;tee &lt;/span&gt;stress-ng.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At the same time, collect system metrics:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C vmstat &lt;span class="nt"&gt;-y&lt;/span&gt; &lt;span class="nt"&gt;-t&lt;/span&gt; 1 120 | &lt;span class="nb"&gt;tee &lt;/span&gt;vmstat.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C mpstat &lt;span class="nt"&gt;-P&lt;/span&gt; ALL 1 120 | &lt;span class="nb"&gt;tee &lt;/span&gt;mpstat.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Production warning:&lt;/strong&gt; Do not deliberately saturate CPU on a production VM unless you have an approved test or maintenance window.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A controlled test is useful because one known variable is introduced:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Predictable CPU workload
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can then observe how the VM and its metrics respond.&lt;/p&gt;




&lt;h2&gt;
  
  
  Compare Like with Like
&lt;/h2&gt;

&lt;p&gt;Do not compare &lt;code&gt;stress-ng&lt;/code&gt; numbers from unrelated systems as if they were equivalent.&lt;/p&gt;

&lt;p&gt;Try to keep the following constant:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;stress-ng&lt;/code&gt; version&lt;/li&gt;
&lt;li&gt;CPU method&lt;/li&gt;
&lt;li&gt;operating-system image&lt;/li&gt;
&lt;li&gt;CPU class&lt;/li&gt;
&lt;li&gt;vCPU count&lt;/li&gt;
&lt;li&gt;CPU limits&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Otherwise, differences may come from the test environment rather than from the platform behavior you are trying to measure.&lt;/p&gt;




&lt;h2&gt;
  
  
  What Can the Hosting Provider See That You Cannot?
&lt;/h2&gt;

&lt;p&gt;Inside the VM, you only see guest-level evidence.&lt;/p&gt;

&lt;p&gt;The infrastructure team may have access to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;vCPU thread wait time&lt;/li&gt;
&lt;li&gt;host oversubscription&lt;/li&gt;
&lt;li&gt;scheduler statistics&lt;/li&gt;
&lt;li&gt;physical CPU utilization&lt;/li&gt;
&lt;li&gt;vCPU pinning&lt;/li&gt;
&lt;li&gt;host events&lt;/li&gt;
&lt;li&gt;hypervisor-level contention&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those measurements are much closer to the actual scheduling layer.&lt;/p&gt;

&lt;p&gt;If possible, ask the provider to compare its host-side data with the exact same UTC interval from your guest logs.&lt;/p&gt;




&lt;h2&gt;
  
  
  A Practical Diagnostic Workflow
&lt;/h2&gt;

&lt;p&gt;Here is a simple sequence you can reuse during an incident.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Measure &lt;code&gt;%st&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Collect interval-based &lt;code&gt;vmstat&lt;/code&gt; data:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C vmstat &lt;span class="nt"&gt;-y&lt;/span&gt; &lt;span class="nt"&gt;-t&lt;/span&gt; 1 60
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 2: Check the run queue
&lt;/h3&gt;

&lt;p&gt;Look at:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;r
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ask whether runnable work is accumulating.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Inspect each vCPU
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C mpstat &lt;span class="nt"&gt;-P&lt;/span&gt; ALL 1 60
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Look for imbalance.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Inspect application threads
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C pidstat &lt;span class="nt"&gt;-u&lt;/span&gt; &lt;span class="nt"&gt;-t&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; PID 1 60
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check whether the guest workload itself is saturating CPU.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5: Check cgroup throttling
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/cpu.stat
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Look for increasing throttling counters.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 6: Align application metrics
&lt;/h3&gt;

&lt;p&gt;Compare the same interval for:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;p95
p99
throughput
error rate
request rate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 7: Eliminate alternative causes
&lt;/h3&gt;

&lt;p&gt;Check:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;I/O&lt;/li&gt;
&lt;li&gt;garbage collection&lt;/li&gt;
&lt;li&gt;locks&lt;/li&gt;
&lt;li&gt;deployments&lt;/li&gt;
&lt;li&gt;network dependencies&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Step 8: Ask for host telemetry
&lt;/h3&gt;

&lt;p&gt;If the symptoms still point toward host CPU scheduling, ask the platform team to inspect the corresponding host interval.&lt;/p&gt;




&lt;h2&gt;
  
  
  How to Write a Useful Support Ticket
&lt;/h2&gt;

&lt;p&gt;Do not start with an accusation.&lt;/p&gt;

&lt;p&gt;This is weak:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Your server is overloaded because another customer is stealing our CPU.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You cannot prove that from guest metrics.&lt;/p&gt;

&lt;p&gt;Instead, write something measurable.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Between 10:14 and 10:22 UTC, &lt;code&gt;%st&lt;/code&gt; increased significantly relative to the normal baseline for this VM. During the same interval, the CPU run queue increased and application p99 latency exceeded the SLO.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That gives the provider a concrete period and measurable symptoms to investigate.&lt;/p&gt;

&lt;p&gt;Include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;VM identifier&lt;/li&gt;
&lt;li&gt;region&lt;/li&gt;
&lt;li&gt;number of vCPUs&lt;/li&gt;
&lt;li&gt;kernel version&lt;/li&gt;
&lt;li&gt;virtualization type&lt;/li&gt;
&lt;li&gt;exact UTC interval&lt;/li&gt;
&lt;li&gt;normal baseline&lt;/li&gt;
&lt;li&gt;observed &lt;code&gt;%st&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;run queue&lt;/li&gt;
&lt;li&gt;p95/p99&lt;/li&gt;
&lt;li&gt;throughput&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Attach your raw data:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;metadata.txt
vmstat.txt
mpstat.txt
application.csv
README.md
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before sending anything, remove:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;passwords&lt;/li&gt;
&lt;li&gt;API keys&lt;/li&gt;
&lt;li&gt;access tokens&lt;/li&gt;
&lt;li&gt;user data&lt;/li&gt;
&lt;li&gt;sensitive request contents&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Repeat the Test After Migration or a Plan Change
&lt;/h2&gt;

&lt;p&gt;Suppose the provider migrates the VM to another host.&lt;/p&gt;

&lt;p&gt;Or you change:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the VM plan&lt;/li&gt;
&lt;li&gt;CPU limits&lt;/li&gt;
&lt;li&gt;vCPU count&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not simply look at the next random workload and declare the problem fixed.&lt;/p&gt;

&lt;p&gt;Repeat a comparable test.&lt;/p&gt;

&lt;p&gt;Keep the following as consistent as possible:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Duration
Input workload
Number of processes
Traffic profile
Measurement interval
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One run is weak evidence.&lt;/p&gt;

&lt;p&gt;Several repeated runs are much stronger.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Before:
%st elevated
p99 elevated
throughput reduced

After migration:
%st lower
p99 restored
throughput restored
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If that pattern repeats across several comparable tests, the case for a platform-related effect becomes much stronger.&lt;/p&gt;




&lt;h2&gt;
  
  
  Monitor Steal Time with Prometheus
&lt;/h2&gt;

&lt;p&gt;If you use &lt;code&gt;node_exporter&lt;/code&gt;, CPU steal time can be observed through:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;node_cpu_seconds_total{mode="steal"}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A five-minute percentage can be calculated with PromQL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100 * avg by (instance) (
  rate(node_cpu_seconds_total{mode="steal"}[5m])
)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This converts the cumulative steal-time counter into an approximate percentage over the selected window.&lt;/p&gt;




&lt;h2&gt;
  
  
  Do Not Alert on a Single Spike
&lt;/h2&gt;

&lt;p&gt;A simplistic alert might look conceptually like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st &amp;gt; X
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But this ignores:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the VM's normal baseline&lt;/li&gt;
&lt;li&gt;duration&lt;/li&gt;
&lt;li&gt;user impact&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A more useful alerting model is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st above expected baseline
AND
condition persists
AND
application impact exists
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Application impact might include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;p99 &amp;gt; SLO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;run queue increased
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;throughput decreased
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;error rate increased
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With Prometheus, the &lt;code&gt;for&lt;/code&gt; clause can prevent a single temporary spike from creating an incident.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10m&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;means the condition must remain true for ten minutes before the alert fires.&lt;/p&gt;

&lt;p&gt;The exact duration should depend on the workload and SLO.&lt;/p&gt;




&lt;h2&gt;
  
  
  Monitor the Monitoring System Too
&lt;/h2&gt;

&lt;p&gt;One subtle problem remains.&lt;/p&gt;

&lt;p&gt;Suppose your dashboard shows:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st = 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That only means something if the metric is actually being collected.&lt;/p&gt;

&lt;p&gt;If the exporter stops reporting the time series entirely, you should not interpret the missing metric as zero.&lt;/p&gt;

&lt;p&gt;A reliable monitoring system should therefore distinguish between:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Steal time is zero
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Steal-time telemetry disappeared
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Monitoring the monitoring pipeline is part of production observability.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Three-Signal Model
&lt;/h2&gt;

&lt;p&gt;A useful way to think about steal-time incidents is to use three layers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. Scheduling signal
        ↓
       %st

2. CPU pressure
        ↓
        r

3. User-visible impact
        ↓
 p95 / p99 / throughput
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The strongest diagnosis appears when all three move together.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st ↑
r ↑
p99 ↑
throughput ↓
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;is strong evidence that CPU scheduling pressure is having a measurable effect.&lt;/p&gt;

&lt;p&gt;By contrast:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%st spike
r unchanged
p99 unchanged
throughput unchanged
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;may not justify any operational response at all.&lt;/p&gt;




&lt;h2&gt;
  
  
  Practical Checklist
&lt;/h2&gt;

&lt;p&gt;When you notice elevated CPU steal time, work through this list.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Collect &lt;code&gt;%st&lt;/code&gt; using interval-based measurements&lt;/li&gt;
&lt;li&gt;[ ] Record UTC timestamps&lt;/li&gt;
&lt;li&gt;[ ] Check the &lt;code&gt;vmstat&lt;/code&gt; run queue&lt;/li&gt;
&lt;li&gt;[ ] Inspect individual vCPUs with &lt;code&gt;mpstat&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;[ ] Inspect application threads with &lt;code&gt;pidstat&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;[ ] Check cgroup CPU throttling&lt;/li&gt;
&lt;li&gt;[ ] Compare p95 and p99 latency&lt;/li&gt;
&lt;li&gt;[ ] Compare throughput&lt;/li&gt;
&lt;li&gt;[ ] Compare error rate&lt;/li&gt;
&lt;li&gt;[ ] Match aggregation windows&lt;/li&gt;
&lt;li&gt;[ ] Compare against the VM's baseline&lt;/li&gt;
&lt;li&gt;[ ] Check I/O and application bottlenecks&lt;/li&gt;
&lt;li&gt;[ ] Preserve raw data&lt;/li&gt;
&lt;li&gt;[ ] Request host telemetry if needed&lt;/li&gt;
&lt;li&gt;[ ] Repeat the test after migration or configuration changes&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Final Takeaway
&lt;/h2&gt;

&lt;p&gt;CPU steal time &lt;code&gt;%st&lt;/code&gt; tells you that a runnable vCPU did not receive physical CPU time during part of a measurement interval.&lt;/p&gt;

&lt;p&gt;That information is useful.&lt;/p&gt;

&lt;p&gt;But it is not a complete diagnosis.&lt;/p&gt;

&lt;p&gt;Do not ask only:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Is &lt;code&gt;%st&lt;/code&gt; high?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Ask instead:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Is &lt;code&gt;%st&lt;/code&gt; persistently abnormal for this VM, and does it coincide with CPU pressure and measurable application degradation?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A solid investigation combines:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;interval-based steal-time measurements&lt;/li&gt;
&lt;li&gt;CPU run queue&lt;/li&gt;
&lt;li&gt;per-vCPU utilization&lt;/li&gt;
&lt;li&gt;cgroup throttling&lt;/li&gt;
&lt;li&gt;application latency&lt;/li&gt;
&lt;li&gt;throughput&lt;/li&gt;
&lt;li&gt;historical baseline&lt;/li&gt;
&lt;li&gt;host telemetry when available&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Without the run queue and application impact, &lt;code&gt;%st&lt;/code&gt; remains an observation.&lt;/p&gt;

&lt;p&gt;With synchronized system and application metrics, it becomes evidence you can actually use.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>performance</category>
      <category>virtualization</category>
    </item>
  </channel>
</rss>
