DEV Community: Da

"A Pod at 40 Percent CPU Can Still Freeze Every 100 Milliseconds"

Da — Sun, 02 Aug 2026 09:28:37 +0000

A Pod at 40 Percent CPU Can Still Freeze Every 100 Milliseconds

A Kubernetes Pod can show only 40 percent CPU usage and still suffer severe CPU throttling because the graph is averaging activity across a much larger time window than the Linux scheduler uses to enforce the limit.

That mismatch is the whole problem.

A dashboard may average CPU usage over ten seconds, thirty seconds, or one minute. Linux can enforce a container CPU quota over a period measured in milliseconds. A bursty workload can consume its entire allowance near the beginning of that short period and then be prevented from running until the quota resets.

The graph later reports a moderate average.

The application experiences a pause.

This is why a latency-sensitive service can have healthy-looking utilization and poor p99 response time at the same moment.

The discussion centered on this exact failure mode. It also exposed a broader debate about whether CPU limits should be removed, retained for predictability, or applied only in specific environments.

The useful conclusion is not “CPU limits are always bad” or “CPU limits are always necessary.” The useful conclusion is that teams need to understand the time model before they trust the percentage.

CPU percentages hide the time dimension

Several participants argued that the easiest way to understand throttling is to stop thinking in percentages and start thinking in CPU time.

A process may use an entire core for a very short period and then remain mostly idle. When that burst is averaged across a long scrape interval, the result can look tiny.

One example in the discussion described an application using one full core for 50 milliseconds and then doing nothing for almost twenty seconds. On a long graphing window, that can appear as a fraction of one percent CPU.

Yet if the container has a very low CPU limit, the burst can still exceed the available quota inside the short CFS period and be throttled.

The percentage is not false. It answers a different question.

It tells you how much CPU time was consumed across the averaging window.

It does not tell you whether the workload exhausted its quota inside one scheduler enforcement period.

That distinction is especially important for modern web services. Many request paths are bursty. They perform a short block of CPU-intensive work, then wait on storage, a database, a remote API, or network I/O.

Average utilization can remain low while request latency is dominated by short scheduler pauses.

CFS quota is enforced in short periods

The discussion repeatedly used a 100 millisecond CFS period to explain the behavior.

A CPU limit is translated into an amount of CPU time available during each period. If the workload consumes that quota early, it cannot run again until the next period begins.

For example, a limit of 100 millicores corresponds conceptually to about 10 milliseconds of CPU time per 100 millisecond period.

If the application needs 2 milliseconds of CPU to serve a request, that may sound comfortably below the limit when averaged over one second.

But timing matters.

If the application needs more CPU time than remains in the current period, the request can be delayed until quota becomes available again. A small amount of actual CPU work can therefore produce a much larger wall-clock latency.

Another example in the discussion used a workload with multiple runnable threads on a four-core node. A container with a quota equivalent to 200 milliseconds of CPU time can consume that quota in only 50 milliseconds of real time if four threads run across four cores simultaneously.

The container then waits for the rest of the period.

Parallelism accelerates quota consumption.

The quota is measured in aggregate CPU time, not simply elapsed wall time.

Low average usage and high throttling are not contradictory

One participant questioned how a Pod at 40 percent CPU could be throttled, assuming throttling should happen only near 100 percent of the limit.

The replies clarified that usage percentage and throttling do not move together in a simple linear relationship.

A workload can show low average usage and still be heavily affected by throttling if its CPU demand arrives in short bursts.

It can also show relatively high average usage without throttling if its limit is large enough and the demand fits within each quota period.

The relationship depends on:

The CPU limit
The CFS period
The number of runnable threads
How quickly CPU demand arrives
The graph averaging window
Whether the node has free CPU
Whether other workloads create contention
The difference between CPU time and wall-clock time

This is why the intuitive model of “50 percent usage means 50 percent throttling” is wrong.

Throttling begins when the cgroup attempts to consume more CPU time than its quota permits during the enforcement period.

The long-term average may still be low.

The averaging trap is worst for latency-sensitive services

The strongest practical comment came from an operator who said their team removed CPU limits from request-path Deployments and saw p99 latency fall immediately. Their nr_throttled metric also dropped to zero.

That is a compelling operational result, but it should be interpreted carefully.

Removing a CPU limit allows the workload to use spare node capacity when it needs to burst. For a latency-sensitive service, this can eliminate quota-induced pauses.

The improvement can be dramatic when the application performs brief CPU-heavy work and then waits on I/O.

The danger is assuming spare CPU will always be available.

Under light cluster load, an unlimited container may burst freely. Under node contention, neighboring workloads also compete for CPU. The service may then receive only the share implied by its request and scheduling weight.

That can cause a different performance surprise.

A service that looked fast during low utilization may slow down during a traffic event precisely because neighboring Pods are also consuming their share.

Removing limits can eliminate CFS quota throttling. It does not create guaranteed burst capacity.

Requests and limits solve different problems

The discussion repeatedly returned to the distinction between CPU requests and CPU limits.

CPU requests influence scheduling and relative CPU entitlement under contention.

CPU limits impose a hard cap enforced through cgroup quota.

These settings should not be treated as interchangeable.

A realistic request tells the scheduler how much capacity the workload needs and influences how CPU is divided when the node is busy.

A limit tells the kernel not to allow the container to consume beyond a defined ceiling, even if spare CPU exists.

This creates two different risk models.

With a low request and no limit, the workload may perform well while the node has spare capacity but lose burst headroom under contention.

With a request equal to the limit, performance becomes more predictable, but bursty work may experience quota throttling if the value is too low.

With an inflated request, the workload may receive strong protection but reduce cluster efficiency.

There is no universal value that solves all three concerns.

Why some operators remove CPU limits

Several commenters argued that CPU limits provide little value for most ordinary workloads and can create avoidable latency.

Their reasoning was practical:

CPU is compressible
The kernel can share CPU between workloads
Requests already influence fair allocation
Hard limits can waste spare node capacity
Bursty applications are vulnerable to quota pauses
Averaged dashboards often hide the effect
Memory limits are more important because memory is not compressible in the same way

This approach is common for trusted application workloads in clusters with good observability and realistic requests.

The principle is to let workloads borrow unused CPU rather than forcing idle capacity to remain unusable.

That can improve utilization and latency at the same time.

It works best when the platform team understands node contention, monitors requests carefully, and can identify noisy neighbors.

It is less comfortable in environments where workloads are untrusted or cost boundaries must be strict.

Why some operators keep CPU limits

The author of the original discussion listed several cases where CPU limits still made sense:

Multi-tenant environments
Workloads from different teams or customers
Untrusted code
Protection against runaway processes
Cost control
Managed platform models that require explicit ceilings

Another participant argued that limits improve predictability.

Their concern was that unlimited burst behavior can deceive operators. A service may look consistently fast because it uses spare CPU that is not guaranteed. Later, when adjacent Pods become busy, that burst capacity disappears and the service slows down during the worst possible traffic event.

From this perspective, a limit exposes the true performance envelope earlier.

If the workload cannot meet its latency objective within the allocated CPU ceiling, the allocation is wrong and should be fixed before production pressure reveals it.

This is a legitimate position.

Predictability has a cost. It may require unused headroom and careful load testing.

The debate is not really about whether unused CPU should exist. It is about whether the organization wants opportunistic performance or guaranteed performance.

For latency-sensitive services, load testing is the deciding tool

One operator argued that latency-sensitive containers should be load tested to determine a reasonable maximum load with headroom, then configured with request and limit set to that value.

This creates a Guaranteed QoS-style allocation and makes performance more deterministic.

The original author agreed that this is the ideal scenario but questioned whether most teams have time to test every application that thoroughly.

That tension is real.

The technically rigorous answer requires workload-specific evidence.

Teams should test:

Normal traffic
Peak traffic
Burst concurrency
Cold starts
Startup CPU demand
Downstream latency
Node contention
Multiple replicas
Autoscaling lag
Different CPU request and limit combinations
Tail latency
Throttling counters

A simple web service may not justify an elaborate performance program.

A critical request path may.

The correct policy can therefore vary by workload class.

Platform teams can provide safer defaults while requiring deeper testing only for services with strict latency or reliability objectives.

The number of threads changes how quickly quota disappears

The discussion included a detailed question about multithreading.

If a container can use several cores at once, it can consume aggregate CPU quota faster in wall-clock time.

A workload with four runnable threads can spend four milliseconds of CPU time during one millisecond of real time if all four threads run concurrently.

That is why a quota equivalent to 200 milliseconds of CPU time can disappear in 50 milliseconds on four cores.

A workload limited to two runnable threads consumes the same quota more slowly. With only two threads, the maximum aggregate consumption over 100 milliseconds of wall time is 200 milliseconds of CPU time.

This means a 2000 millicore limit can accommodate two fully busy threads without quota exhaustion under the simplified example.

The important variable is not only the CPU limit. It is how much parallel CPU time the application can consume during the period.

Languages and runtimes with worker pools, parallel garbage collection, or many runnable threads can therefore hit quota behavior differently from single-threaded services.

`kubectl top` is not enough

The discussion strongly criticized relying only on averaged CPU dashboards.

kubectl top and common monitoring panels are useful for broad utilization trends. They are not sufficient for diagnosing CFS throttling.

The source recommended examining cpu.stat, especially:

nr_throttled
throttled_usec

These counters reveal whether the cgroup was prevented from running because it exhausted quota.

A rising nr_throttled value shows that throttling occurred across scheduling periods.

throttled_usec shows accumulated throttled time.

The exact interpretation requires care, especially across multiple threads, but these counters provide evidence that ordinary utilization graphs miss.

The best workflow is to correlate them with application symptoms:

p95 and p99 latency
Request rate
Error rate
Queue depth
CPU usage
Replica count
Node saturation
Garbage collection
Runtime thread behavior

A throttling counter without latency impact may not justify immediate action.

Latency degradation without throttling may point somewhere else.

PSI detects pressure that quota metrics miss

The original post also highlighted Pressure Stall Information through cpu.pressure.

PSI answers a different question from cpu.stat.

cpu.stat can show quota throttling for a cgroup.

CPU pressure can show that tasks are waiting for CPU because the system is under contention, even when no cgroup quota was exhausted.

This distinction matters after removing CPU limits.

A container with no hard CPU limit will not experience quota throttling in the same way, but it can still wait because the node is busy.

That wait can damage latency.

PSI helps reveal system-wide or cgroup-level contention that ordinary utilization averages may not explain clearly.

A useful diagnosis separates at least three cases:

Quota throttling caused by a CPU limit
CPU contention caused by neighboring workloads
Hypervisor delay caused by the virtual machine not receiving physical CPU time

Each requires a different fix.

Hypervisor steal time is another invisible failure mode

The post also mentioned hypervisor steal time, commonly observed as %st.

Steal time represents CPU time the virtual machine wanted but the hypervisor did not provide because the physical host was running something else.

This is not Kubernetes CFS throttling.

It can still look like unexplained application slowness.

A Pod may have reasonable requests, no CPU limit, no concerning throttling counters, and still experience poor performance because the underlying VM is not receiving expected CPU time.

That is why cpu.stat is necessary but not sufficient.

The diagnosis should extend down the stack:

Application latency
Container throttling
cgroup pressure
Node CPU contention
Hypervisor steal
Instance type behavior
Burstable VM credits where applicable

Kubernetes performance is often a layered scheduling problem.

The application runtime, Linux kernel, cgroups, Kubernetes scheduler, and cloud hypervisor can all delay execution for different reasons.

QoS class does not eliminate bad CPU sizing

A participant asked how Kubernetes QoS classes fit into the discussion.

The thread did not provide a complete answer, but it exposed the relevant principle.

Setting requests equal to limits can place a workload into a more predictable resource configuration, but that does not make the numbers correct.

A Guaranteed-style allocation with a CPU limit that is too low can still throttle badly.

A Burstable workload with a realistic request and no CPU limit may perform better under light load but vary more under contention.

QoS class is therefore not a substitute for capacity testing.

It influences eviction behavior and resource treatment, but it does not automatically determine whether the application can meet its latency objective.

The values still need to match the workload.

CPU limits do not protect a service from bad requests

One commenter pointed out that a CPU limit would not save an important service if its request were too low and neighboring Pods consumed the remaining CPU.

That correction is important.

A limit protects the node and other workloads from one container consuming beyond its ceiling.

It does not reserve additional CPU for the limited container.

The request is the important value for scheduling and guaranteed share under contention.

If a latency-sensitive service depends on spare CPU but has a tiny request, removing the limit may improve performance in quiet periods while leaving it exposed during pressure.

A robust design therefore begins with realistic requests.

The request should reflect the workload’s real minimum capacity requirement, not a long-term average chosen only to improve cluster density.

Accurate requests are harder than they look

The discussion argued that most teams set CPU requests incorrectly in one of two directions.

They set them too high for safety.

Or they set them near a long-term average that is too low for bursts.

Both are understandable.

High requests waste schedulable capacity.

Low requests create a false sense of performance when the node is quiet and unpredictable latency when contention arrives.

This is why requests should be informed by:

Load tests
Traffic profiles
High-percentile usage
Concurrency
Threading behavior
Latency objectives
Autoscaling behavior
Node contention tests
Growth expectations

Average CPU alone is a weak sizing signal.

For latency-sensitive workloads, CPU time per request and concurrency can be more useful.

Queueing theory explains why predictability costs money

One operator summarized the tradeoff bluntly: teams cannot have perfect efficiency and perfect predictability at the same time.

If incoming demand is not fully predictable, the system needs spare capacity to absorb variation.

That is a queueing problem.

High utilization increases the probability that new work must wait.

This applies at the application thread pool, container CPU allocation, node capacity, and cluster level.

Removing CPU limits may improve opportunistic use of spare capacity, but it does not eliminate the need for headroom.

Setting limits and requests to a tested maximum can improve predictability, but the cluster may run with more unused CPU.

The correct choice depends on the business value of latency stability versus infrastructure efficiency.

The mistake is pretending the tradeoff does not exist.

A practical troubleshooting path starts below the dashboard

When a service shows unexplained latency despite moderate CPU graphs, the investigation should move through the stack deliberately.

First, confirm whether a CPU limit exists.

Then inspect the workload’s request and limit values.

Check cpu.stat for throttling counters.

Correlate those counters with latency.

Inspect CPU pressure to identify contention.

Check node-level saturation and neighboring workloads.

Review hypervisor steal time.

Examine thread count and runtime behavior.

Test the service under a controlled load.

Finally, repeat the test with adjusted limits or no limit, depending on the environment.

The goal is to identify the specific delay mechanism before applying a generic policy.

The right CPU policy is workload-specific

The discussion produced several competing recommendations:

Remove CPU limits for latency-sensitive request paths
Keep realistic requests
Use limits in multi-tenant or untrusted environments
Set request equal to limit after load testing
Treat predictable performance as worth the unused capacity
Prefer no limits for ordinary trusted workloads
Monitor memory requests and limits carefully
Use direct cgroup metrics instead of dashboard averages

These positions are not fully compatible as universal rules.

They become compatible when applied to different workload classes.

A platform policy can distinguish:

Trusted internal services
Public request-path services
Batch workloads
Multi-tenant workloads
Untrusted code
Strict-cost workloads
Safety-critical workloads
Development environments

Each class can have a different default and testing requirement.

That is more mature than enforcing one CPU rule across the entire cluster.

FAQ

Can a Pod be throttled while showing only 40 percent CPU?

Yes. The graph may average CPU over seconds or minutes while CFS enforces quota over a much shorter period. A burst can exhaust the quota and cause a pause that disappears inside the average.

What causes CFS CPU throttling?

A container is throttled when it consumes the CPU time allowed by its cgroup quota during the current enforcement period.

Why do multiple threads matter?

Multiple runnable threads can consume CPU time across several cores simultaneously, exhausting aggregate quota faster in wall-clock time.

Which metrics show actual throttling?

The discussion recommends inspecting nr_throttled and throttled_usec in cpu.stat and correlating them with application latency.

What does CPU pressure show?

PSI through cpu.pressure can reveal time tasks spend waiting for CPU because of contention, including cases where no CPU quota is configured.

Should latency-sensitive Pods have CPU limits?

The discussion did not reach one universal answer. Some operators remove limits to eliminate quota pauses, while others prefer tested request-equals-limit settings for predictability.

When are CPU limits most useful?

The source discussion identified multi-tenant clusters, untrusted workloads, runaway-process protection, cost control, and managed platform environments as common cases.

Is a realistic CPU request still important without a limit?

Yes. Requests influence scheduling and CPU share under contention. A low request can leave a service vulnerable when neighboring workloads become busy.

CTA

CPU performance problems often hide below averaged dashboards, across cgroups, node pressure, and virtualization layers. Request an online trial and explore how Sensaka helps teams correlate infrastructure signals with real application behavior.

"Cl0p Targets PTC Windchill: Why Product Lifecycle Systems Need Better Backup Planning"

Da — Sun, 02 Aug 2026 09:09:49 +0000

PTC has warned customers about a critical remote code execution vulnerability in Windchill and FlexPLM, tracked as CVE-2026-12569. The company says an unauthorized user may be able to execute code remotely and has published patches, indicators of compromise, and hunting guidance. Censys later connected active exploitation to Cl0p affiliates and observed fewer than 100 internet exposed Windchill instances in early June 2026, with most identified systems located in the United States. The small exposed population does not reduce the potential impact. Product lifecycle management platforms can hold engineering documents, bills of materials, supplier data, product changes, and intellectual property that support manufacturing and regulated workflows. Organizations should apply the current PTC guidance immediately, but patching alone cannot prove that a previously exposed system is clean. Recovery planning should begin with the distinction explained in backup versus disaster recovery: backups preserve recoverable data, while disaster recovery restores complete services, dependencies, identities, and operating processes.

A Patch Stops a Vulnerability but Does Not Erase an Intrusion

PTC’s advisory includes webshell paths, suspicious request patterns, command and control addresses, and recommendations for searching logs and filesystems. Those details matter because an attacker who exploited the vulnerability before patching may have established persistence, stolen credentials, modified application files, or moved into connected systems. Installing the fixed version closes the known path but does not remove every artifact created earlier. Affected organizations need to preserve evidence, review the vendor’s current indicators, inspect identity and administrative activity, and determine when unauthorized access may have begun. This timeline controls which backups can be trusted. A recovery point created after compromise may contain the same webshell, malicious configuration, or stolen credential material that caused the incident.

Windchill Recovery Requires More Than Application Files

A product lifecycle system usually depends on databases, file vaults, search indexes, application servers, identity services, certificates, integrations, and workflow configuration. Restoring only the visible document repository may leave the platform inconsistent or unusable. Recovery teams should document the order in which components are restored, how database and file states are synchronized, which external systems must be available, and how users are validated after recovery. The disaster recovery plan framework provides a useful structure for service priorities, dependencies, roles, contact paths, recovery objectives, infrastructure rebuilds, communications, and testing. Windchill owners should adapt that framework to their exact architecture instead of assuming the software team can reconstruct the environment during an emergency.

Clean Recovery Points Need Independent Protection

Cl0p has repeatedly targeted enterprise software that sits close to valuable data. Once attackers gain administrative access, online backups and connected storage may also be exposed. Organizations should maintain copies in separate failure domains, restrict backup administration, and use immutability or offline media where appropriate. The 3-2-1-1-0 backup rule adds an offline or immutable copy and requires zero unverified backup errors. That final condition is important because a successful job status does not confirm that application data is consistent, complete, or free from attacker persistence. Retention should also extend far enough to reach a point before the earliest credible compromise date.

Recovery Testing Should Include a Compromise Scenario

A useful test does not restore Windchill into the same trusted network and immediately reconnect every integration. It restores the platform into an isolated environment, checks the selected recovery point for known indicators, validates database and file consistency, rotates relevant credentials, and confirms that application workflows function before production access resumes. Teams should measure how long investigation, rebuild, restoration, validation, and user acceptance actually take. Cl0p’s Windchill activity demonstrates that vulnerability management and backup operations cannot remain separate disciplines. Patching reduces future exposure, while clean, isolated, and tested recovery provides a path forward when exploitation may already have happened.

Originally published on the Mr.PlanB blog.

"HollowFrame and Matryoshka Malware Target Law Firms Through Spear Phishing"

Da — Sun, 02 Aug 2026 09:08:40 +0000

Security researchers disclosed a multistage intrusion involving a previously undocumented Go based loader called HollowFrame and a Rust based backdoor tracked as Matryoshka. According to the July 31, 2026 report, the attack targeted two endpoints at an unnamed law firm. The initial spear phishing message linked to an encrypted archive containing a Windows shortcut disguised as “Case Documents.” Opening the shortcut triggered a chain involving privilege escalation, weakened Microsoft Defender protections, persistence, and additional payload delivery. The lure is significant because it fits normal legal work. Lawyers and staff routinely receive matter documents, links, archives, and files from clients, opposing counsel, experts, courts, and vendors. A familiar label can therefore bypass skepticism before technical security controls have a chance to respond. Law firms should connect endpoint defense with the broader duty described in MIRA’s guide to attorney client confidentiality, which extends across communications, files, calendars, billing records, and technology workflows.

The Attack Separated Malicious Activity Across Stages

HollowFrame operated as a modular loader and persistence framework. Researchers reported that it used legitimate software components alongside malicious files, checked whether it was running in an analysis environment, and created a scheduled task to remain active. Matryoshka then provided remote command execution and additional capabilities. One observed variant communicated through conventional web traffic, while another used a private GitHub repository for command delivery, results, and file transfer. Dividing the intrusion into stages reduced the amount of suspicious behavior visible in any single component. That design can make detection and attribution harder because a security product may see only one part of the full chain. For a law firm, persistent remote access could expose matter files, credentials, email, identity systems, and information about clients whose work is not publicly known.

Confidentiality Risk Extends Beyond Privileged Documents

A compromise does not need to reach a privileged legal memorandum to create serious harm. Attorney client confidentiality generally covers a broader range of client related information than evidentiary privilege. Matter names, contact records, calendars, research history, billing narratives, access logs, and file metadata can reveal relationships, strategy, timing, or business activity. MIRA’s comparison of privilege and confidentiality explains why firms must protect operational records as well as traditional legal communications. This matters in a spear phishing case because attackers often use the first compromised endpoint to identify additional users, systems, and valuable matters. Even limited initial access can support credential theft, lateral movement, or more convincing messages sent from a trusted account.

Law Firms Need Controls Around Normal Document Work

The attack chain depended on a recipient opening an archive and executing a shortcut that appeared related to a case. Firms should restrict risky attachment and archive types, make file extensions visible, prevent ordinary users from bypassing endpoint protection, and control script execution where practical. Email security, endpoint detection, identity monitoring, least privilege, application controls, and tested incident response should work together. Training should use realistic legal examples rather than generic phishing simulations, because staff need to recognize how attackers imitate matter documents, court notices, signature requests, invoices, and client communications. When a suspicious file is reported, the response should examine related accounts and systems rather than treating the event as an isolated email.

Vendor and Workflow Reviews Must Include Security

Legal technology selection often concentrates on usability and integration, while security review arrives late in the process. That creates blind spots when a product accesses email, documents, calendars, client information, or billing systems. MIRA’s legal timekeeping software checklist includes security, confidentiality, integrations, administration, and implementation among the factors firms should evaluate. The same discipline should apply to every application that touches matter data. HollowFrame and Matryoshka demonstrate that attackers do not need an exotic legal specific exploit. They can enter through an ordinary document workflow, hide activity across several stages, and use familiar cloud services to support command and control. Strong protection begins with recognizing that routine legal work is itself part of the attack surface.

Originally published on the MIRA News and Blog.

"Boston Law Firm Raises Starting Salaries to $235,000: What It Means for BigLaw Competition"

Da — Sun, 02 Aug 2026 09:08:04 +0000

McDermott Will & Schulte has raised first year associate salaries to $235,000, becoming the first Boston law firm to publicly announce that level, according to Boston.com. The increase took effect on July 1, 2026 and follows a broader compensation move led by Milbank, which raised its associate scale from $235,000 for first years to as much as $455,000 for eighth year associates. For law students and junior lawyers, the immediate effect is straightforward: a higher market benchmark gives candidates another way to compare firms. For management, the impact is more complicated. Salary increases affect recruiting, retention, billing rates, leverage, utilization expectations, and the amount of work each lawyer must convert into collected revenue. A firm can afford a higher compensation scale when client demand, rates, realization, and productivity support it. That makes accurate time capture more commercially important, especially when the cost of each professional hour is rising. MIRA’s guide to missed billable hours explains how delayed entry, fragmented work systems, and forgotten tasks can quietly reduce the revenue available to support compensation.

Compensation Is Becoming a Market Signal

BigLaw salaries communicate more than pay. They signal the type of matters a firm expects to handle, the clients it wants to serve, and the talent market in which it intends to compete. A Boston office matching a leading national scale may attract candidates who would otherwise choose New York, Washington, or another major legal market. It can also create pressure on peer firms that recruit from the same schools and practice areas. However, matching the salary does not automatically create the same economics. Firms differ in billing rates, client mix, partner leverage, realization, office costs, and the volume of premium work available to junior associates. A compensation decision that strengthens one firm’s recruiting position may compress another firm’s margins if its underlying demand is weaker.

Higher Salaries Increase the Cost of Time Leakage

A few missing entries can appear minor at the individual level, but the effect compounds across a large associate population. When lawyers reconstruct time days later, they often omit short calls, document review, research, messages, and transitions between matters. The firm still pays the salary and overhead associated with that work, but some of the revenue opportunity disappears. MIRA’s Timekeeping AI Calculator is designed to translate small daily omissions into an estimated annual financial effect. The calculation is directional rather than a substitute for firm financial analysis, yet it helps leaders see why time capture is connected to compensation strategy. As salaries rise, the value of improving capture by even a small percentage can become material.

Productivity Cannot Mean Unreviewed Billing

Pressure to recover more time must be balanced with professional judgment, client guidelines, and clear descriptions. Firms should not respond to higher compensation by pushing every detected activity directly onto an invoice. Time entries still need matter matching, lawyer review, confidentiality controls, accurate task classification, and compliance with billing rules. The better approach is to reduce forgotten work while preserving human approval. MIRA’s guide to legal timekeeping best practices recommends prompt entry, specific matter related descriptions, consistent coding, review before release, and practical technology that supports daily habits. That combination can improve revenue quality without weakening client trust.

The Competition Will Extend Beyond Base Salary

Candidates evaluate bonuses, training, partnership prospects, flexibility, workload predictability, culture, and the quality of matters alongside salary. Firms that raise pay without improving the associate experience may still struggle with retention. At the same time, firms that do not match the highest scale can compete through clearer career development, specialized practices, reasonable expectations, or stronger long term opportunities. The $235,000 benchmark will matter most where firms are pursuing the same candidates and clients. It also places more attention on the operating systems behind profitability. Compensation announcements are public, but the ability to support them depends on private disciplines such as demand forecasting, matter management, billing quality, time capture, realization, and client retention.

Originally published on the MIRA News and Blog.

"VMware to Proxmox Migration: Lessons From Centrogest’s Three Node Cluster"

Da — Sun, 02 Aug 2026 09:07:03 +0000

Centrogest’s move from VMware to Proxmox VE offers a useful example of why a virtualization migration should begin with architecture rather than license replacement. A Hardware Upgrade report on the project says AXOL Server designed a three-node Proxmox cluster with distributed NVMe storage, redundant 10 GbE networking, a separate Proxmox Backup Server, and an offsite copy protected by Object Lock. The previous platform had redundant compute nodes, but its data remained on one aging storage system. That made storage the critical weakness. The project therefore addressed performance, resilience, recovery, and power-loss behavior together. This is the central lesson in any Proxmox and VMware comparison: the hypervisor matters, but migration outcomes depend on the complete operating model around it.

The Migration Solved a Storage Problem First

Centrogest’s former environment had developed through successive additions and no longer matched the company’s growth. The VMware cluster provided compute redundancy, while data remained concentrated on a single system built with aging 10,000-rpm mechanical disks. AXOL Server assessed the workloads and then sized compute, memory, storage, networking, backup, and power protection as one design. The new cluster uses three Proxmox nodes, each with three 8 TB NVMe drives. The report describes 72 TB of raw capacity with three-way replication, leaving approximately 24 TB available to virtual machines after redundancy. The design also includes 768 GB of installed memory and redundant 10 GbE switches. The important planning principle is that raw capacity does not equal usable capacity. Replication, failover headroom, performance requirements, and maintenance states all reduce what can be safely assigned to workloads.

Backup Was Built as a Separate Recovery Layer

The cluster protects against a node failure, but Centrogest did not treat cluster redundancy as a backup. A separate physical server runs Proxmox Backup Server with local storage, encryption, deduplication, and extended retention. Selected data is then copied to S3-compatible storage with Object Lock, creating an offsite layer intended to remain protected if the primary environment is compromised. The design also includes file-level backup, local Microsoft 365 protection, and replicated volume snapshots. Each mechanism addresses a different failure mode. Administrators planning a similar project should review the capabilities and operational requirements of Proxmox Backup Server, then define recovery objectives before selecting retention, immutability, and offsite capacity. A backup architecture is complete only when teams can restore the required service within an agreed time.

Power and Operations Belong in the Migration Plan

The project also connects the UPS to the Proxmox environment through Network UPS Tools. When battery capacity falls below a defined threshold, the system begins an orderly sequence that shuts down virtual machines before the cluster nodes. That detail matters because platform migrations often focus heavily on conversion tools and overlook facilities, monitoring, shutdown order, documentation, and support ownership. Centrogest retained responsibility for application priorities while AXOL Server handled assessment, architecture, configuration, migration, and production deployment. This division made technical responsibilities explicit. A structured VMware migration assessment should likewise inventory dependencies, storage behavior, network paths, backup coverage, application owners, test criteria, and rollback options before workloads move. Centrogest’s case shows that a successful VMware exit is an infrastructure redesign with measurable recovery and availability goals, not a one-for-one hypervisor swap.

Originally published on the Mr.PlanB blog.

Why AI Data Centers Need More Than GPU Monitoring

Da — Sat, 01 Aug 2026 10:45:14 +0000

GPUs are the most visible and expensive resources in an AI data center, so it is reasonable that operations teams begin by monitoring them.

Utilization, memory use, temperature, power, clocks, and error counters reveal important information about accelerator health and activity. NVIDIA Data Center GPU Manager provides extensive telemetry and diagnostic capabilities for exactly this purpose.

But an AI workload does not run on a GPU alone.

It depends on containers, physical nodes, high speed networks, storage systems, data pipelines, power, cooling, schedulers, projects, users, and service objectives. Monitoring only the accelerator can show that performance has changed. It cannot consistently explain what changed around it or what the business impact will be.

This is why mature AI data center operations need a broader model.

GPU monitoring answers a narrow set of questions

GPU telemetry is good at answering questions such as:

Is the accelerator active?
How much memory is being used?
What are the temperature and power readings?
Are clock or thermal limits affecting performance?
Are ECC or hardware error conditions present?

These questions matter. The NVIDIA DCGM documentation shows how GPU management includes health, diagnostics, configuration, power profiles, and operating state.

However, operations teams are usually asked broader questions:

Why did this training job slow down?
Which workload is using this GPU?
Is the problem caused by network communication or data delivery?
Which project and service will be affected if this node fails?
Is cooling capacity sufficient for the current rack density?
Which team is holding idle capacity?
How much GPU consumption belongs to each tenant or project?

These questions require context outside the accelerator.

AI workloads span multiple infrastructure domains

A training or inference request passes through several layers before useful work is completed.

A scheduler assigns resources. A container starts on a node. Training data arrives from storage. GPUs exchange information through a high speed fabric. The server draws power and rejects heat through air or liquid cooling. Results are written back to storage or delivered through a model service.

A constraint in any of these layers can reduce performance or cause failure.

NVIDIA's current guidance states that AI data center observability must handle high volume accelerator telemetry, diverse high speed networks, significant data throughput, and correlated signals across large GPU environments. Its telemetry and observability guidance describes unified visibility as a requirement for performance and service objectives at scale.

The operational lesson is direct: GPU monitoring should remain part of the system, but it should not define the boundary of the system.

Workload context explains who is using the GPU

A GPU metric has limited meaning without workload context.

An accelerator may show low utilization because the assigned job is waiting for data. It may show high memory consumption because a container reserved more memory than it actively uses. It may appear idle because the training phase has moved to evaluation or checkpointing.

Operations teams need to connect:

Task to container
Container to GPU
GPU to physical node
Node to cluster and resource pool
Workload to project, tenant, model, and owner

This relationship model makes it possible to move from a device symptom to the responsible workload and team.

Without it, every investigation begins with manual questions: Who owns this process? Which project requested the resource? Is the job still useful? What service depends on it?

A live AI infrastructure CMDB provides this relationship context by connecting facilities, devices, accelerators, workloads, services, projects, and owners.

Network monitoring explains communication waits

Distributed training depends on synchronized communication across GPUs and nodes. A small network problem can have a large workload effect because workers may wait for the slowest communication path before continuing.

NVIDIA's NCCL documentation describes multi GPU and multi node communication as a foundation for efficient training scale. Its troubleshooting guide directs teams to investigate interface selection, fabric health, latency, bandwidth, InfiniBand, RoCE, and topology when performance degrades.

Relevant network signals include:

Packet loss and retransmission
Port errors and link state
Latency and bandwidth
Congestion and uneven traffic
GPU to NIC topology
Collective communication duration
One slow worker delaying the group

A GPU utilization chart can show the waiting. Network telemetry can explain it.

Storage monitoring explains data starvation

GPUs need a consistent supply of training data. If storage throughput falls or read latency rises, accelerators may wait even though the workload is active.

Storage also affects checkpointing, model loading, dataset preparation, and inference response paths. The required bandwidth varies by model, data format, batch size, access pattern, and number of GPUs.

Useful signals include:

Read and write throughput
IOPS
Latency
Queue depth
Data loading time
Checkpoint write duration
Client level imbalance

A complete performance view places these signals beside GPU activity and workload timing. This helps operators distinguish a compute limitation from a data delivery limitation.

Physical hardware monitoring finds faults above and below the operating system

Kubernetes, operating system, and application monitoring are necessary, but they do not replace physical infrastructure visibility.

A server can remain online after losing redundancy. One power supply may fail while the second continues operating. A disk may enter a degraded state without immediately stopping the workload. ECC events, temperature changes, firmware conditions, fan issues, and memory faults may appear before a full node failure.

Out of band monitoring through BMC, Redfish, or IPMI provides an independent view of hardware status. It can remain available when the operating system is unresponsive or the production network is unavailable.

This layer matters because AI workloads are costly to interrupt. A hardware condition that seems minor at the server level can force a long running training job to restart or leave multiple GPUs idle.

Cooling monitoring protects performance and capacity

High density AI racks connect computing performance directly to facility conditions.

GPU temperature may be visible within accelerator monitoring, but operators also need to understand the cooling system around it. In liquid cooled environments, relevant signals can include coolant inlet and outlet temperature, pressure difference, flow, pump or CDU status, leak events, rack power density, and available cooling capacity.

Monitoring these conditions helps answer two different questions:

Is a current cooling condition affecting workload performance or reliability?
Can the facility safely support additional compute density?

This is why liquid cooling monitoring should be connected to the wider AI operations view rather than treated as a separate facilities dashboard.

Usage metering explains efficiency and accountability

A GPU can be online and allocated without being productively used.

Operations teams need to compare capacity assigned with capacity consumed. This is important for shared enterprise platforms, research environments, public computing centers, and any organization where teams compete for limited accelerators.

A practical metering model may include:

Allocated GPU hours
Active and idle periods
Accelerator type
Project and tenant ownership
Workload or model association
Memory allocation and actual use
Energy or infrastructure context where available

These records support showback, internal cost allocation, capacity planning, and better scheduling decisions.

GPU usage metering adds an economic and governance layer that pure GPU monitoring does not provide. It helps answer who consumed the capacity, whether it was used effectively, and where future investment is justified.

Observability requires correlation, not more isolated dashboards

Adding separate tools for GPU, network, storage, cooling, and assets can increase visibility while still leaving the investigation fragmented.

The real operational value comes from aligning signals around the same event and workload.

For example, when training performance drops, a team should be able to review:

GPU utilization and memory
Container allocation and workload phase
Network latency, loss, and retransmission
Storage throughput and data loading time
Node temperature, power, ECC, and hardware events
Cooling conditions
Project, service, and owner relationships

When these signals share a timeline and relationship model, teams can build an evidence based investigation path.

This is the role of AI infrastructure observability. It extends monitoring across compute, network, storage, workload, hardware, power, and cooling so the team can explain why a metric changed, not simply report that it changed.

AI data center operations also need action and accountability

Monitoring has limited value if it ends at a dashboard.

An operational system should help teams decide and record what happens next. Depending on the issue, the response may include:

Isolating a faulty node
Rescheduling a workload
Adjusting resource allocation
Reviewing a network path
Correcting a storage bottleneck
Escalating a cooling condition
Opening a service ticket
Routing the issue to the responsible owner
Recording the decision and change for audit

The response should also consider business impact. A fault affecting an experimental job does not have the same priority as a fault affecting a production inference service.

Connecting monitoring to CMDB relationships, workflows, approvals, and service context turns technical visibility into operations management.

A broader operating model for AI infrastructure

A complete AI data center management platform should bring together five forms of visibility:

Resource visibility

What GPUs, NPUs, servers, networks, storage systems, racks, power, and cooling resources exist, and what is their current condition?

Workload visibility

Which jobs, containers, models, and services are using those resources?

Relationship visibility

How are facilities, devices, resources, workloads, projects, users, and services connected?

Performance visibility

Where are compute, communication, data delivery, hardware, or cooling constraints reducing useful output?

Operational visibility

Who owns the issue, what is affected, what action was approved, and whether the resource or service recovered?

GPU monitoring remains essential within this model. It simply becomes one connected source of evidence rather than the entire operating view.

The shift from monitoring accelerators to operating AI infrastructure

The first generation of AI infrastructure management often begins with a simple requirement: show whether the GPUs are healthy and busy.

As the environment grows, that question becomes insufficient. Teams must understand why workloads slow down, how infrastructure domains interact, what a failure affects, who owns the resources, and whether capacity is delivering useful value.

That shift does not make GPU monitoring less important. It makes context more important.

The organizations that operate AI infrastructure effectively will be the ones that connect accelerator telemetry with workloads, networks, storage, hardware, cooling, relationships, costs, and operational action.

Originally published on the Sensaka blog.

The Hidden Network and Storage Bottlenecks Behind Slow AI Training

Da — Sat, 01 Aug 2026 10:44:38 +0000

When an AI training job slows down, the GPU is usually the first place teams look.

That response is understandable. GPUs are the most visible and expensive part of the training environment. Their utilization, memory use, temperature, and power consumption are easy to chart, and low accelerator activity appears to provide a simple explanation for poor performance.

But the GPU is often waiting rather than failing.

Training depends on a continuous chain of data movement and synchronization. Data must travel from storage into memory, pass through preprocessing, reach the accelerator, and then move between GPUs and nodes during distributed communication. A delay at any point can reduce useful computation while the job remains active.

This is why adding more GPUs does not always make training faster. When the data path or communication fabric is already constrained, additional accelerators may simply create more devices waiting for the same bottleneck.

Slow training is a system problem

A training step involves more than model computation.

Depending on the workload, the system may need to:

Read files or objects from storage
Decompress and decode data
Apply transformations or augmentation
Move batches through CPU and host memory
Transfer data to GPU memory
Run forward and backward computation
Exchange gradients between workers
Synchronize progress across nodes
Write logs, outputs, and checkpoints

The GPU can only work when the preceding stages deliver data on time and the communication stages allow the workload to continue.

A low utilization chart therefore shows the result of waiting. It does not identify what caused the wait.

A useful investigation must connect accelerator activity with workload timing, network behavior, storage performance, CPU pressure, and physical node health. That is the central purpose of AI infrastructure observability.

Distributed training amplifies small network problems

A network issue that seems minor at the infrastructure level can have a large effect on a synchronized training workload.

In data parallel training, multiple workers process batches and exchange gradients. The workers then synchronize before moving to the next stage. If one node communicates more slowly, the other workers may complete their local computation and wait.

This creates a familiar pattern:

The training job remains active
GPU utilization falls across several workers
One node or path shows longer communication time
Step duration increases
Overall throughput declines

The cluster average may hide the original source. Most GPUs may be healthy, but the workload proceeds at the speed of the slowest participant.

Packet loss and retransmission consume training time

High speed AI networks are designed to move large volumes of data with low latency. Even so, packet loss, retransmission, congestion, or unstable links can reduce the effective performance available to distributed jobs.

Operators should compare GPU activity with signals such as:

Packet loss
Retransmission
Port errors
Link state changes
Queue pressure
Latency
Effective bandwidth
Traffic imbalance
Collective communication duration

A short increase in retransmission may not look severe on a network dashboard. If it occurs during a communication intensive phase, however, it can extend every synchronization cycle.

The effect compounds across a long training run. A small delay repeated thousands of times can become hours of lost accelerator productivity.

Network topology affects communication efficiency

The physical and logical path between GPUs, network adapters, switches, and nodes matters.

Two workers may have similar compute capacity but different communication performance because their traffic crosses different paths. A topology mismatch, incorrect interface choice, oversubscribed uplink, or uneven placement can make one group of accelerators consistently slower than another.

Useful questions include:

Are the affected GPUs connected through the expected network adapters?
Are workers distributed across different switch paths?
Does one node have a slower or unstable link?
Are traffic volumes balanced across interfaces?
Did workload placement change before performance declined?
Does the problem affect one job, one rack, or the whole cluster?

These questions require more than network metrics. They require a relationship model that connects the workload to its containers, nodes, GPUs, interfaces, and physical location.

An AI infrastructure CMDB provides that context by mapping dynamic workload relationships alongside physical infrastructure dependencies.

Storage can starve otherwise healthy GPUs

Training workloads frequently read large datasets, many small files, or a mixture of metadata and sequential data. The storage system must deliver those inputs at a rate that keeps the accelerator pipeline full.

If the storage path cannot keep up, the GPU completes its current batch and waits for the next one.

Common symptoms include:

GPU utilization alternating between bursts and idle periods
Data loading time increasing
Read latency rising during busy periods
Storage throughput reaching a ceiling
IOPS saturation
CPU workers waiting on file access
Uneven performance between training nodes

The workload may still appear healthy from the scheduler's perspective. No process has crashed, and no device is offline. The job is simply progressing more slowly than expected.

Throughput alone does not explain storage performance

A storage dashboard may show high throughput and still fail to explain poor training performance.

Different workloads create different access patterns. A model reading a small number of large files behaves differently from one reading millions of small objects. Metadata operations, queue depth, random access, caching, and client imbalance can all affect data delivery.

A complete storage investigation should consider:

Read and write throughput
IOPS
Read and write latency
Queue depth
Metadata response time
Cache behavior
Client level performance
Data loader duration
Dataset and file layout
Concurrent workload demand

High aggregate throughput may coexist with poor latency for one client. Cluster averages can therefore conceal the experience of the individual worker delaying the training job.

Checkpointing creates another storage pressure point

Long running training workloads periodically write checkpoints so progress can be recovered after interruption.

Checkpointing may temporarily reduce GPU activity because the application must serialize and write a large amount of state. A predictable utilization valley can be normal. The operational question is whether that valley remains stable.

Teams should compare:

Checkpoint start and completion times
Write throughput
Write latency
Storage queue pressure
GPU activity during the checkpoint
Changes in checkpoint duration over time

If each checkpoint takes longer than the previous one, the environment may be experiencing contention, capacity pressure, or degradation.

Checkpoint analysis is especially important because a storage problem can affect both current performance and future recovery. A delayed or failed checkpoint increases the amount of work that may need to be repeated after an incident.

Data preprocessing can look like a storage problem

Not every data delivery bottleneck originates in the storage system.

The input pipeline may spend significant time decoding images, transforming records, tokenizing text, decompressing data, or moving batches through host memory. If CPU capacity or memory bandwidth is insufficient, the GPU may wait even when storage performance appears normal.

This is why operators should align:

Storage latency and throughput
CPU utilization
Memory pressure
Data loader worker activity
Batch preparation time
GPU utilization
Training step duration

The purpose is to identify where waiting begins.

A storage metric viewed alone may suggest healthy performance. A workload timeline may reveal that preprocessing consumes most of the interval between batches.

Shared infrastructure creates hidden contention

AI clusters commonly support several teams and workload types at the same time.

A training job may compete with:

Another training workload reading the same dataset
A checkpoint write from a different project
Model deployment activity
Dataset preparation
Backup or replication traffic
Inference services with latency requirements
General infrastructure operations

This contention may be temporary and difficult to reproduce. It may also affect only one storage pool, network segment, rack, or time window.

Useful analysis therefore includes ownership and workload context. Teams need to know which projects were using the same infrastructure when the slowdown occurred.

The Sensaka AI Data Center Management Platform connects infrastructure, workloads, projects, resources, and operational processes so performance problems can be investigated in context rather than as isolated metrics.

Average metrics hide stragglers

One of the most common diagnostic mistakes is relying on cluster averages.

Imagine seven workers completing a training step in roughly the same time while one worker takes significantly longer. The average network latency or storage throughput may still look acceptable. Yet every synchronization point waits for the slow worker.

Operators should compare metrics by:

GPU
Node
Container
Network port
Storage client
Rack
Workload
Project

The goal is to determine whether all workers degrade together or whether one participant becomes a straggler.

When only one worker is affected, investigate its network path, storage client behavior, CPU pressure, GPU health, temperature, power, and recent configuration changes.

Correlation is more useful than dashboard switching

Teams often have access to all the necessary metrics but still struggle to identify the cause.

The problem is fragmentation.

GPU data may live in one tool, network telemetry in another, storage performance in a third, and workload records in the scheduler. Operators manually compare timestamps and attempt to reconstruct what happened.

This process is slow and vulnerable to missed relationships.

A stronger approach aligns the following signals on one timeline:

GPU utilization and memory
Training step duration
Data loading time
Collective communication time
Network latency, loss, and retransmission
Storage throughput, IOPS, and latency
CPU and memory activity
Hardware health and temperature
Checkpoint events
Workload placement and ownership

The purpose is not to collect the largest possible number of metrics. It is to connect the signals that explain one workload's behavior.

A practical troubleshooting sequence

When training slows down, teams can use a structured sequence instead of opening every monitoring tool at once.

1. Confirm the workload pattern

Determine when the slowdown began, whether it is continuous or periodic, and whether it affects one phase or the entire job.

Review recent changes to the workload, dataset, cluster placement, and resource specification.

2. Compare workers

Review GPU utilization, step duration, communication time, and data loading time by worker.

Identify whether the whole job slows down together or one node becomes the straggler.

3. Inspect network behavior

Check latency, packet loss, retransmission, port errors, link state, traffic imbalance, and communication duration.

Map the affected worker to its network interface and path.

4. Inspect storage behavior

Review throughput, IOPS, latency, queue depth, client performance, and checkpoint duration.

Compare the affected worker with healthy workers reading the same data.

5. Inspect preprocessing

Review CPU, memory, data loader workers, decoding, transformations, and batch preparation time.

Determine whether the input pipeline is blocked before data reaches the GPU.

6. Check physical infrastructure

Review GPU temperature, power, ECC events, node hardware status, and cooling conditions.

A physical node problem can reduce network, storage, or compute performance without causing an immediate outage.

7. Review contention and ownership

Identify other workloads sharing the same network or storage resources.

Determine whether scheduling, placement, or capacity policies should change.

The goal is useful GPU time

GPU availability is not the same as GPU productivity.

A cluster may have sufficient accelerator capacity and still deliver poor training throughput because communication and data delivery cannot sustain the workload.

This is why AI infrastructure planning must treat compute, network, and storage as one operating system. Capacity in one domain has limited value when another domain prevents that capacity from producing useful work.

The most effective teams do not stop when they discover low GPU utilization. They trace the waiting time across the full infrastructure path, identify the constraint, and connect the finding to workload ownership and operational action.

That approach turns a slow training job from a vague GPU problem into an evidence based infrastructure diagnosis.

Originally published on the Sensaka blog.

Why RMM Tools Cannot Replace Physical Infrastructure Monitoring

Da — Sat, 01 Aug 2026 08:26:46 +0000

Remote monitoring and management tools are widely used to support distributed IT environments.

They help teams monitor operating systems, install software, apply patches, run scripts, collect performance data, and provide remote support.

For endpoints, branch servers, and managed service environments, these capabilities are valuable.

The problem begins when organizations assume that RMM visibility is equivalent to physical infrastructure visibility.

It is not.

RMM tools usually see the system through the operating system or an installed agent. Data center operations must also see what exists below, beside, and around the operating system.

That includes:

Hardware components
Power supplies
Fans
Disks
Memory errors
Management controllers
Rack location
Power paths
Cooling
Temperature
UPS systems
PDUs
Network infrastructure
Storage hardware
Facility dependencies

An RMM platform can be part of the operating model.

It cannot replace the full set of physical infrastructure monitoring capabilities a data center requires.

RMM and physical infrastructure monitoring solve different problems

RMM tools are usually designed around device administration.

Common functions include:

Agent based monitoring
CPU and memory metrics
Disk utilization
Process monitoring
Service monitoring
Patch management
Software inventory
Remote desktop
Script execution
Endpoint security
Ticket integration
User support

Physical infrastructure monitoring focuses on a different set of questions:

Is the hardware healthy?
Is redundancy available?
Where is the device located?
Which power path supports it?
Is the rack within power limits?
Is cooling adequate?
Which component is failing?
Can the device be reached when the operating system is down?
Which services depend on the physical infrastructure?

The guide comparing RMM and DCIM explains why these categories overlap in some areas but serve different operational objectives.

The operating system is not the hardware

An operating system can report useful hardware information.

It may expose:

CPU
Memory
Disk
Network interfaces
Temperatures
Device drivers
Storage errors

However, the view is incomplete.

The operating system may not reliably report:

Power supply redundancy
Fan health
Controller battery condition
Hardware event logs
Preboot failures
Management controller state
Firmware inventory
Chassis intrusion
Voltage conditions
Physical drive health
Hardware sensor changes
Power consumption
Inlet temperature
Component replacement history

It may also stop reporting when the server is:

Powered off
Hung
Booting
Reinstalling
Experiencing kernel failure
Disconnected from the production network
Affected by agent failure
Running an unsupported operating system

The hardware continues to exist even when the operating system cannot communicate.

Physical monitoring must remain available in those conditions.

Agent failure can look like device failure

RMM tools often depend on an installed agent.

The agent may stop reporting because of:

Service crash
Certificate expiration
Network change
Firewall rule
Operating system upgrade
Permission change
Resource exhaustion
Software conflict
Corrupted installation
Device reboot

When this happens, the platform may report the endpoint as offline.

That does not explain whether the cause is:

Agent failure
Operating system failure
Network failure
Power failure
Hardware failure
Planned maintenance

Independent infrastructure monitoring provides another source of evidence.

For example, an out of band management controller may still report:

Power state
Hardware health
Event log
Temperature
Network reachability
Console access

This helps the team determine what layer has failed.

Hardware redundancy can be lost while the system remains online

One of the most important physical monitoring problems is degraded redundancy.

Examples include:

One of two power supplies has failed
One disk in a RAID group has failed
One network path is unavailable
One fan has failed
One storage controller is offline
One UPS module is unavailable
One cooling unit is under maintenance

The application may continue running.

The operating system may appear healthy.

The RMM dashboard may remain green.

The infrastructure has become more vulnerable, however.

A second failure may create an outage.

Physical monitoring should identify:

Which redundancy was lost
What protection remains
Which service is exposed
How long repair may take
Whether the condition is worsening

This is a different question from whether the operating system is online.

Out of band monitoring sees below the production layer

Out of band monitoring uses a management path that is independent of the normal operating system and business network.

Common interfaces include:

BMC
iLO
iDRAC
IMM
iBMC
Redfish
IPMI
Vendor management APIs

These interfaces may provide:

Power state
Sensor data
Component inventory
Hardware event logs
Remote console
Virtual media
Power control
Firmware information
Health status

The guide to out of band monitoring explains how this independent management path supports monitoring and recovery when the operating system or production network is unavailable.

RMM remote access is usually dependent on the operating system.

Out of band access can remain available before the operating system starts and after it fails.

Remote desktop is not the same as remote console

RMM platforms often provide remote desktop or terminal access.

This is useful when the operating system is running.

A remote console through the management controller supports different situations.

It may allow an operator to:

Watch the boot process
Enter BIOS or firmware setup
Review startup errors
Mount installation media
Reinstall the operating system
Diagnose a failed boot
Change low level configuration
Restart a frozen server

These capabilities are important during severe incidents.

When the operating system cannot start, RMM remote access is usually unavailable.

The difference is operationally significant.

Physical location matters during incidents

RMM tools commonly organize devices by:

Customer
Site
Group
Department
Operating system
Policy

Data center teams also need physical location.

This may include:

Building
Room
Row
Rack
U position
Chassis
Blade slot
Power connection
Network port
Cable path

Physical location helps teams:

Find failed equipment
Dispatch technicians
Replace components
Plan rack capacity
Trace power
Trace network connections
Manage moves
Support audits
Identify local environmental risk

A hostname alone is not enough when someone must locate and repair the device.

RMM tools do not model rack power and cooling

RMM platforms can collect server power or temperature in some environments.

They usually do not model the full facility relationship.

Data center monitoring may need to connect:

Device power
Rack power
PDU capacity
Circuit capacity
Redundant feeds
UPS load
Generator capacity
Cooling zone
Inlet temperature
Hot aisle
Cold aisle
Room conditions

This supports questions such as:

Can another server be added to this rack?
Is the rack close to its power limit?
Are both power feeds balanced?
Which devices are affected by a PDU failure?
Is one rack creating a hotspot?
Does cooling capacity match IT load?
Is redundancy available during maintenance?

These are physical infrastructure questions.

They are outside the normal scope of RMM.

Environmental conditions can affect healthy operating systems

A server may appear healthy while the environment becomes unsafe.

Examples include:

Rising room temperature
Local rack hotspot
High humidity
Water leak
Smoke
Cooling failure
UPS battery issue
PDU overload
Utility disturbance

The operating system may continue reporting normal CPU and memory metrics until the condition becomes severe.

Facility monitoring can provide earlier warning.

The data center monitoring software guide describes the broader monitoring scope required across IT equipment, hardware health, network, power, cooling, and environmental systems.

Storage and network hardware require specialist visibility

RMM tools are usually strongest on endpoints and operating systems.

Data centers also contain:

Storage arrays
Fibre Channel switches
Network switches
Routers
Load balancers
Firewalls
Backup appliances
Hyperconverged platforms
Tape libraries
Security appliances

These devices may expose:

Controller health
Disk groups
Cache state
Port errors
Optical power
Fabric topology
Routing state
Fan health
Power supply state
Capacity
Latency

RMM may monitor basic availability or receive selected metrics.

It usually does not provide the full cross vendor physical and operational depth required for these platforms.

A complete data center view must include more than Windows, Linux, and endpoint agents.

Physical assets need lifecycle management

Data center equipment has a physical lifecycle.

It is:

Purchased
Delivered
Accepted
Installed
Configured
Monitored
Maintained
Upgraded
Moved
Retired
Disposed

Lifecycle data may include:

Serial number
Component configuration
Warranty
Support contract
Rack location
Power connection
Maintenance history
Failure history
Replacement
Disposal evidence

RMM asset inventory often focuses on software and operating system information.

Physical asset management requires evidence that connects procurement, configuration, location, health, maintenance, and retirement.

BMC monitoring provides a separate hardware evidence source

The baseboard management controller is an independent subsystem inside many servers.

It can monitor hardware even when the main operating system is unavailable.

A BMC may provide:

Sensor readings
Fan state
Power supply state
Temperature
Voltage
Hardware logs
Power control
Remote console
Firmware
Component information

The glossary page on BMC monitoring explains why this management layer is important for hardware operations.

RMM data and BMC data can complement each other.

For example:

RMM shows operating system performance
BMC shows physical component health
Application monitoring shows user experience
Network monitoring shows connectivity
DCIM shows power, cooling, and location

Together, these layers provide stronger evidence.

Physical monitoring supports failure analysis

When a service fails, teams need to determine which layer caused the problem.

Possible layers include:

Application
Database
Operating system
Virtualization
Network
Storage
Server hardware
Power
Cooling

RMM can help investigate operating system and application conditions.

It may not show:

A failed disk in the storage array
A power feed problem
A degraded network path
A hardware controller reset
A failed server fan
A rack temperature issue

Without this evidence, engineers may spend time investigating software symptoms.

A layered monitoring model shortens fault isolation.

RMM is valuable for distributed operations

The limitation of RMM should not be confused with lack of value.

RMM platforms are useful for:

Endpoint management
Patch management
Remote support
Software deployment
Script execution
Policy enforcement
Operating system monitoring
User device administration
Distributed site support

They are often essential in managed service environments and large endpoint estates.

The issue is scope.

A tool designed for endpoint and operating system management should not be expected to provide complete physical infrastructure intelligence.

DCIM does not replace RMM either

The relationship works both ways.

DCIM and physical infrastructure monitoring usually do not replace all RMM functions.

They may not provide:

End user remote support
Patch management
Software deployment
Endpoint policy
User session troubleshooting
Desktop management
Application installation

A mature architecture uses tools according to their strengths.

For example:

RMM for endpoint and operating system administration
Hardware monitoring for component health
DCIM for physical assets, capacity, power, and cooling
Network monitoring for network depth
Application monitoring for user experience
ITSM for incident and change workflows

The objective is coordinated operations, not forcing one product to perform every role.

Integration matters more than replacement

Instead of asking whether RMM should replace DCIM or hardware monitoring, organizations should ask how the systems should work together.

Useful integrations include:

Shared asset identity
Alert forwarding
Incident creation
Ownership
Service relationships
Remote action links
Change records
Reporting
Lifecycle updates

For example, an operating system alert from RMM can be enriched with:

Physical rack location
Hardware health
Power status
Warranty
Business service
Recent changes

This gives the operator more context without requiring one platform to collect everything.

Use a layered monitoring model

A useful monitoring architecture can be organized into layers.

Layer 1: Business and user experience

Monitor:

Transactions
Availability
Response time
User journeys
Service level objectives

Layer 2: Applications and databases

Monitor:

Processes
Requests
Errors
Queries
Connections
Application dependencies

Layer 3: Operating systems and virtualization

Monitor:

CPU
Memory
Disk
Processes
Services
Virtual machines
Clusters

Layer 4: Network and storage

Monitor:

Ports
Traffic
Errors
Paths
Controllers
Volumes
Latency
Capacity

Layer 5: Hardware

Monitor:

Fans
Power supplies
Disks
Memory errors
Controllers
Firmware
Sensors

Layer 6: Facilities

Monitor:

Power
UPS
PDU
Cooling
Temperature
Humidity
Water
Fire
Physical location

RMM usually covers part of layers two and three.

Physical infrastructure monitoring covers layers four, five, and six.

Evaluate the monitoring gap directly

Organizations can assess whether RMM coverage is sufficient by testing practical scenarios.

Ask whether the current toolset can answer:

Which server has lost power redundancy?
Which rack is close to its power limit?
Which physical disk is failing?
Which devices depend on a failed PDU?
Can we access a server that will not boot?
Which equipment is affected by rising inlet temperature?
Where is the device physically located?
Which component changed after maintenance?
Which servers have expiring warranty?
Which network path is degraded?
Which service depends on the affected hardware?

If these questions cannot be answered, the environment has a physical visibility gap.

Measure the operational value of deeper monitoring

Physical infrastructure monitoring should be evaluated through outcomes.

Useful measures include:

Hardware warnings detected before outage
Mean time to identify failed components
Mean time to locate equipment
Number of incidents requiring site travel
Time to recover inaccessible servers
Number of devices operating without full redundancy
Asset accuracy
Warranty coverage accuracy
Rack power visibility
Thermal incidents
Manual inspection time

These measures show whether the added monitoring layer improves reliability and efficiency.

Tool consolidation should not remove necessary evidence

Organizations often want fewer tools.

This can reduce cost and complexity.

However, consolidation should not remove critical visibility.

Before retiring a platform, verify:

Which data it uniquely collects
Which actions it uniquely supports
Which teams depend on it
Which incidents require it
Whether replacement coverage is proven
Whether historical data will remain available
Whether integrations are ready

A smaller toolset is useful only when the remaining systems still provide the evidence needed for safe operations.

The right question is not RMM or DCIM

RMM and physical infrastructure monitoring are not direct substitutes.

They address different operational layers.

RMM helps teams manage operating systems, software, and remote endpoints.

Physical monitoring helps teams understand hardware, power, cooling, location, redundancy, and facility risk.

Data center reliability depends on both logical and physical evidence.

The strongest operating model connects them.

When a service fails, the team should be able to move from user impact to application, operating system, network, storage, hardware, power, and cooling without losing context.

That is the level of visibility a data center requires.

Originally published on the Sensaka blog.

Where AIOps Delivers Real Value in Data Center Operations

Da — Sat, 01 Aug 2026 08:26:10 +0000

AIOps is often presented as a broad promise.

The platform will detect anomalies, predict failures, reduce alerts, identify root causes, automate decisions, and improve service reliability.

Those claims sound useful, but they are too general to guide investment.

Data center teams need a more practical question:

Where does AIOps create measurable operational value?

The answer is usually found in a small number of recurring problems:

Too many alerts
Too many disconnected tools
Slow fault isolation
Limited understanding of business impact
Repeated manual analysis
Poor use of historical data
Reactive maintenance
Inconsistent operational decisions

AIOps creates value when it improves one of these outcomes in a way that can be measured.

It should help the team make a better decision, make it sooner, or execute it with less effort.

AIOps begins with usable operational data

AI cannot compensate for missing, inconsistent, or unreliable data.

AIOps depends on inputs such as:

Metrics
Events
Logs
Traces
Topology
Configuration
Asset data
Change records
Incidents
Workload patterns
Environmental data
Hardware health
Business service relationships

The quality of the result depends on whether these sources describe the same infrastructure consistently.

For example, one system may identify a server by hostname, another by serial number, and another by IP address. If those identities are not reconciled, the platform may treat one device as several unrelated objects.

The guide to what AIOps is provides a useful foundation for understanding how machine learning, event analysis, topology, and automation fit into IT operations.

Before evaluating algorithms, organizations should verify that they can answer basic data questions:

What is the object?
Where is it?
Who owns it?
What service depends on it?
What changed recently?
Which data source is authoritative?
How current is the data?

Without this foundation, AIOps can produce sophisticated looking conclusions that operators do not trust.

Alert reduction is often the first practical use case

Large infrastructure environments generate many alerts.

The volume grows because:

Several tools monitor the same device
One failure creates many downstream symptoms
Static thresholds generate repeated events
Maintenance activity is not suppressed
Short lived conditions create noise
Different tools use different severity levels
Alerts are not linked to ownership
Duplicate events are not removed

AIOps can help by:

Grouping duplicates
Correlating related events
Suppressing known maintenance
Identifying recurring patterns
Prioritizing by service impact
Reducing repeated notifications
Detecting likely initiating events

The value is not simply a lower alert count.

The real value is that operators spend less time reading noise and more time addressing the conditions that matter.

Useful measures include:

Alerts per device
Alerts per incident
Duplicate alert rate
Percentage of alerts requiring action
Average acknowledgment time
Number of missed critical alerts
Operator time spent on triage

A platform that reduces volume but hides important events has not improved operations.

The goal is higher signal quality.

Correlation helps teams see one incident instead of many symptoms

Infrastructure failures often produce a chain of alerts.

A network switch failure may create:

Device unreachable events
Server connectivity alarms
Storage path warnings
Application timeout alerts
Synthetic monitoring failures
Database connection errors

Without correlation, different teams may investigate each symptom separately.

AIOps can use time, topology, dependency, and event similarity to identify that these alerts are probably related.

The result should be one operational incident with:

Likely initiating event
Affected assets
Affected services
Supporting evidence
Current owner
Recommended next checks

This reduces duplicated effort and shortens the path to diagnosis.

However, event correlation should remain explainable.

Operators need to understand why alerts were grouped and which evidence supports the conclusion.

A black box that hides events without explanation will struggle to gain trust.

Anomaly detection is most useful where static thresholds fail

Static thresholds are easy to understand.

For example:

CPU above 90 percent
Temperature above a defined limit
Latency above a fixed value
Disk usage above 80 percent

These rules are useful, but they do not adapt to changing workload patterns.

A value may be normal for one device and abnormal for another. A short spike may be harmless. A small but sustained shift may be an early warning.

AIOps can help detect:

Deviation from normal behavior
Unusual combinations of metrics
Gradual performance drift
Repeating abnormal patterns
Seasonal changes
Peer group differences
Sudden changes after maintenance
Behavior that differs from expected workload

For example, a storage system may remain below a static latency threshold while its response time increases steadily over several days.

An anomaly model may identify the change earlier because it compares current behavior with the system's own baseline.

The article on AIOps anomaly detection approaches provides a useful comparison of methods and the tradeoffs between sensitivity, explainability, and operational usefulness.

Anomalies should be connected to risk

Not every unusual condition matters.

AIOps should help distinguish among:

Unusual but harmless
Unusual and worth watching
Unusual and likely to affect service
Unusual and requiring immediate action

This requires context.

The platform may need to consider:

Business criticality
Current redundancy
Capacity margin
Recent changes
Historical incident patterns
Maintenance schedule
Time of day
Workload importance
Failure probability
Recovery time

For example, a temperature increase in an unused lab rack is different from the same increase in a rack supporting a high value transaction service.

Anomaly detection creates value when it helps the team understand operational consequence.

Root cause analysis is valuable when it narrows the search

Automated root cause analysis is one of the most attractive AIOps promises.

In complex environments, however, a platform may not be able to prove one exact cause.

A more realistic and useful objective is to narrow the search.

AIOps can combine:

Topology
Event timing
Metric changes
Logs
Configuration
Service dependencies
Historical incidents
Recent changes

It can then produce a ranked set of likely causes.

For example:

Network interface errors increased on a core switch.
Storage path failures followed.
Database latency rose.
Application response time degraded.
User transactions began failing.

This sequence gives the team a useful starting point.

The system does not need to be perfect to create value. It needs to reduce the number of systems and possibilities that engineers must investigate manually.

Useful measures include:

Mean time to identify the responsible domain
Mean time to isolate the likely cause
Number of teams involved
Number of tools opened
Number of false root cause suggestions
Percentage of incidents with useful ranking

Change correlation often produces immediate value

Many incidents follow a change.

Examples include:

Firmware upgrade
Network configuration change
Application deployment
Storage migration
Security policy update
Hardware replacement
Capacity adjustment
Cloud configuration change

Operations teams often ask:

What changed?

The answer may require checking several systems.

AIOps can improve this process by linking incidents to:

Approved changes
Configuration differences
Deployment records
Device replacements
Policy updates
Topology changes
Capacity changes

This can shorten diagnosis significantly.

For example, if packet loss begins minutes after a switch configuration update, the platform should highlight that relationship.

Change correlation is valuable because it uses data organizations often already have.

The challenge is ensuring that change records are complete and accurately linked to the affected infrastructure.

Predictive maintenance works best with clear failure patterns

Predictive maintenance is often described as the ability to forecast hardware failure before it occurs.

This is possible in some situations, but the quality depends on available evidence.

Useful inputs may include:

Error rate
Temperature trend
Fan speed
Power supply state
Memory error frequency
Disk media errors
Controller events
Battery condition
Port error growth
Performance degradation
Device age
Failure history
Peer group behavior

The platform may detect that a component is becoming less reliable before it fails completely.

This creates time to:

Move workloads
Schedule maintenance
Order parts
Verify warranty
Test failover
Replace equipment
Reduce load

Predictive maintenance creates the most value when:

The failure mode has measurable precursors
Data is collected frequently
Historical failures are available
The device population is large enough for comparison
The organization has a process for acting on the prediction

A prediction without ownership and workflow has limited operational value.

Hardware intelligence extends AIOps below the operating system

Many AIOps programs focus on application, operating system, cloud, and network data.

Data center operations also need hardware level signals.

These may include:

Fan health
Power supply redundancy
Disk state
Memory errors
Controller health
Firmware
Temperature
Voltage
Hardware event logs
Out of band management data

A service can remain available while its hardware resilience is already reduced.

For example, one power supply may fail while the second continues operating. A RAID array may be degraded while applications remain online.

AIOps can help connect these low level signals to service risk.

This is especially important in AI infrastructure, where high density systems, accelerators, cooling, power, and network dependencies interact closely.

Capacity forecasting should support decisions, not just trends

Capacity forecasting is another useful AIOps use case.

The platform may analyze historical growth and predict when a resource will reach a limit.

Examples include:

Rack power
Cooling capacity
Storage
Network bandwidth
Memory
Compute
GPU allocation
Floor space
Port capacity

A forecast becomes valuable when it supports an action.

For example:

Which rack will reach its power limit first?
When should storage expansion begin?
Which network link needs upgrade?
Can a new cluster be deployed safely?
Which capacity is stranded?
Which service is growing faster than expected?

Forecasts should include uncertainty and assumptions.

A single precise date can create false confidence.

Useful output may show:

Expected range
Growth scenario
Current trend
Seasonal effect
Confidence
Recommended action date
Lead time

Business impact analysis improves prioritization

Infrastructure severity does not always match business severity.

A failed component may have no immediate user impact because redundancy is available.

A small performance change may affect a critical revenue service.

AIOps can improve prioritization by connecting infrastructure conditions to business services.

This may include:

Service criticality
Dependency map
User impact
Transaction volume
Revenue relevance
Geographic scope
Recovery priority
Remaining redundancy

The guide to business service intelligence explains how infrastructure and operational data can be organized around business services.

This context helps answer:

Which incident should be handled first?
Which team should lead?
Which change is safest?
Which users are affected?
What is the likely business consequence?

Automated recommendations reduce dependence on individual memory

Experienced operators often solve incidents by recognizing familiar patterns.

They remember:

Which alerts usually occur together
Which device model has a known issue
Which command reveals the problem
Which sequence of actions is safe
Which vendor article applies
Which team should be called

This knowledge may exist only in individual memory.

AIOps can help capture and reuse it through:

Knowledge bases
Historical incident analysis
Runbooks
Recommended checks
Suggested remediation
Similar incident retrieval
Vendor guidance
Operational policies

The platform may recommend:

Check power redundancy.
Review the hardware event log.
Verify recent firmware changes.
Confirm service failover.
Open a vendor case.
Schedule component replacement.

Recommendations should be evidence based and linked to the current situation.

Operators should be able to review and approve them.

Automation should begin with low risk actions

AIOps is often connected to automation and intelligent agents.

The safest approach is progressive.

Early automation may include:

Collect diagnostics
Enrich an incident
Assign ownership
Open a ticket
Notify the responsible team
Gather recent changes
Run a read only check
Update status
Create a report

Later automation may include:

Restart a service
Move a workload
Change a configuration
Reboot a device
Adjust capacity
Apply a remediation

Higher risk actions require:

Approval
Role control
Prechecks
Maintenance windows
Rollback
Audit
Post action validation

The best first use cases are repetitive, well understood, and easy to verify.

AIOps value depends on workflow integration

A model can identify an important condition, but the organization still needs to act.

AIOps should connect with:

Incident management
Change management
Maintenance
Capacity planning
Asset management
Automation
Collaboration
Reporting

For example, a predicted disk failure may trigger:

Asset identification
Warranty lookup
Service impact check
Workload migration review
Maintenance ticket
Replacement task
Closure validation

This is where operational value appears.

The AI result becomes part of a controlled process.

Trust requires explanation

Operators are responsible for production systems.

They need to understand why the platform made a recommendation.

Useful explanations may include:

Metrics that changed
Related alerts
Historical comparison
Similar past incidents
Dependency path
Recent changes
Confidence score
Alternative causes
Data sources

A recommendation such as “replace the network card” is weak without supporting evidence.

A stronger output might explain:

Packet errors increased by 400 percent
The interface reset three times
Similar devices did not show the same behavior
Application latency rose after each reset
No recent network configuration change was found

Explainability improves trust and helps engineers verify the result.

Measure operational outcomes, not AI activity

AIOps programs sometimes report:

Number of models
Number of events processed
Number of data sources
Number of predictions
Number of automations

These metrics describe activity.

They do not prove value.

Better measures include:

Reduction in duplicate alerts
Reduction in triage time
Reduction in mean time to identify
Reduction in mean time to resolve
Increase in incidents linked to services
Increase in warnings resolved before outage
Reduction in manual diagnostic steps
Reduction in repeated incidents
Increase in capacity forecast accuracy
Operator adoption
Percentage of recommendations accepted
Percentage of automated actions completed safely

The article on AIOps use cases provides a broader set of practical scenarios that can be connected to measurable outcomes.

Choose a narrow first use case

AIOps programs often fail when they begin with an ambition to transform all operations at once.

A better approach is to choose one high value problem.

Examples include:

Reduce duplicate infrastructure alerts
Correlate hardware and application incidents
Predict storage capacity limits
Detect network anomalies
Identify lost hardware redundancy
Link incidents to recent changes
Recommend diagnostic steps
Prioritize incidents by business impact

The first use case should have:

Available data
Clear ownership
Repeated occurrence
Measurable baseline
Low implementation risk
Operational importance

Success can then support expansion.

AIOps does not replace operational discipline

AIOps cannot fix unclear ownership, missing processes, or unreliable asset data on its own.

Organizations still need:

Defined responsibilities
Accurate inventory
Service relationships
Change control
Incident process
Maintenance procedures
Capacity planning
Security controls
Human review

AI can accelerate these processes.

It cannot eliminate the need for them.

The strongest value appears at decision points

AIOps creates real value when it changes what the team does.

Examples include:

Ignore this alert because it is a duplicate
Escalate this warning because redundancy is lost
Investigate this network device first
Delay this deployment because capacity is insufficient
Replace this component before the next maintenance window
Move this workload before hardware failure
Route this incident to the correct team
Prioritize this issue because a critical service is affected
Execute this approved runbook automatically

These are operational decisions.

They are concrete, measurable, and connected to outcomes.

AIOps should make operations simpler

The purpose of AIOps is not to add another layer of complexity.

It should reduce the amount of manual work required to understand and manage complex infrastructure.

The strongest use cases usually involve:

Better signal quality
Faster correlation
Earlier risk detection
Narrower root cause search
Better business prioritization
Reusable operational knowledge
Controlled automation

AIOps delivers value when operators can say:

We saw the right problem sooner, understood it faster, and acted with less effort.

That is a much stronger result than simply saying that AI was added to the monitoring platform.

Originally published on the Sensaka blog.

How to Manage Multi Vendor Infrastructure Without Creating More Silos

Da — Sat, 01 Aug 2026 08:07:03 +0000

Enterprise infrastructure is rarely built from one vendor.

A typical environment may contain servers from several manufacturers, storage from multiple generations, network equipment from different suppliers, virtualization platforms, public cloud services, security appliances, databases, operating systems, and specialized business applications.

Each technology often arrives with its own management tool.

The result is predictable:

Separate dashboards
Different terminology
Duplicate alerts
Conflicting inventory
Separate credentials
Multiple support processes
Different data retention
Repeated integrations
Isolated teams

Organizations often respond by adding another tool intended to provide a unified view.

If that platform simply copies data into one more database without improving ownership, workflow, and service context, it becomes another silo.

The challenge is not to eliminate every specialist tool.

The challenge is to create a common operational layer that allows specialist systems to work together.

Multi vendor infrastructure is the normal enterprise condition

Vendor diversity exists for practical reasons.

Organizations may choose different suppliers because of:

Cost
Performance
Availability
Procurement policy
Regional support
Regulatory requirements
Acquisition history
Workload needs
Existing skills
Vendor risk
Technology cycles

Even organizations with a preferred vendor usually retain older generations, acquired systems, and specialized platforms.

A realistic operating model should therefore assume heterogeneity.

The guide to multi vendor infrastructure management explains why consistent visibility and control matter more than forcing artificial standardization.

Vendor tools are useful but locally optimized

Vendor management systems usually provide deep visibility into their own equipment.

They may support:

Hardware health
Firmware
Configuration
Diagnostics
Capacity
Performance
Remote control
Support integration
Recommended actions

This depth is valuable.

The limitation is that each tool sees only part of the environment.

A storage platform may understand its controllers, disks, volumes, and ports. It may not understand the application slowdown caused by a network event upstream.

A server management tool may report a failed fan. It may not show which business services depend on that server.

A cloud console may show resource utilization. It may not show the physical network path or on premises database dependency.

Specialist tools answer technology questions.

Enterprise operations must also answer cross technology questions.

A single dashboard does not automatically create unified operations

A common mistake is to define unification as putting many widgets on one screen.

A dashboard can display:

Server status
Storage capacity
Network utilization
Cloud spend
Environmental alarms
Application health

This can improve visibility, but it does not guarantee operational integration.

Questions remain:

Are assets matched consistently?
Are duplicate alerts removed?
Is ownership clear?
Can teams follow one incident workflow?
Are service relationships visible?
Are changes recorded?
Can users move from summary to source detail?
Are actions audited?
Is historical data comparable?
Can the platform explain business impact?

Unified operations require common meaning, not only common presentation.

Begin with shared operational outcomes

Before selecting technology, define what the organization wants to improve.

Possible outcomes include:

Reduce time spent switching tools
Shorten fault isolation
Create a consistent asset inventory
Reduce duplicate alerts
Standardize incident routing
Improve cross team collaboration
Understand business impact
Simplify reporting
Improve capacity planning
Support multi site operations
Reduce vendor dependency
Improve audit evidence
Consolidate overlapping tools

These outcomes help determine which data and workflows must be shared.

Without them, integration projects can become technical exercises that produce more connectors but little operational improvement.

Build a common data model

Different tools describe the same infrastructure in different ways.

One tool may identify a server by hostname. Another uses serial number. A network system uses IP address. A facilities system uses rack position. A service platform uses configuration item ID.

A common model should connect these identities.

Core objects may include:

Site
Room
Rack
Device
Component
Network interface
Power connection
Virtual machine
Storage volume
Application
Business service
Owner
Incident
Change

The model does not need to copy every field from every source.

It should define the shared facts required for cross domain operations.

Examples include:

What is this object?
Where is it?
Who owns it?
What is its current state?
What does it depend on?
Which services depend on it?
When did it change?
Which system is authoritative?

This common model allows events from different tools to refer to the same operational object.

Define authoritative sources by data type

A unified platform should not assume that it owns every field.

Different systems may be authoritative for different information.

For example:

Procurement owns purchase and contract data
Hardware tools own component health
Network management owns port and topology data
DCIM owns rack and power relationships
Cloud platforms own cloud resource state
CMDB owns service relationships
ITSM owns incidents and changes
Identity systems own users and groups

The operating model should define:

Which system owns each field
How often data is synchronized
What happens when sources disagree
Which system can create or retire records
How uncertainty is shown
How changes are audited

This prevents the unified platform from becoming another uncontrolled copy of existing data.

Normalize terminology without removing useful detail

Vendors describe similar conditions differently.

Examples include:

Critical
Major
Warning
Degraded
Failed
Predictive failure
Nonrecoverable
Attention required

They may also use different names for components and metrics.

Normalization should map these terms into a common operational language.

For example:

Normal
Warning
Degraded redundancy
Failed
Unreachable
Maintenance
Unknown

However, the original vendor detail should remain available.

Operators may need the precise event code, sensor name, firmware message, or recommended action during diagnosis.

The unified layer should simplify the first response while preserving the path to specialist depth.

Separate collection from interpretation

Multi vendor environments require many collection methods.

These may include:

APIs
SNMP
Redfish
IPMI
Command line access
Syslog
Webhooks
Agents
Out of band interfaces
Cloud APIs
Database queries
File imports

Collection answers:

What data can be retrieved?

Interpretation answers:

What does the data mean operationally?

A platform may successfully collect thousands of metrics while failing to identify which conditions matter.

The operating model should define:

Health rules
Severity
Redundancy state
Ownership
Business criticality
Maintenance windows
Escalation
Service impact

This converts heterogeneous telemetry into consistent operations.

Consolidate alerts before consolidating tools

Organizations often try to remove tools before understanding what those tools contribute.

A safer first step is to unify event handling.

This includes:

Collect alerts
Normalize severity
Remove duplicates
Correlate related events
Suppress maintenance noise
Apply ownership
Add service context
Route incidents
Track acknowledgment
Record resolution

If several tools report the same underlying failure, the operations team should see one actionable incident with supporting evidence.

For example, a failed switch may produce:

Device unavailable alerts
Server connectivity alerts
Storage path alerts
Application timeout alerts
Synthetic monitoring failures

A unified event model should identify the likely initiating condition and show the affected dependencies.

The guide to why monitoring tools miss business critical problems examines how isolated monitoring views can generate data without providing enough service context.

Preserve domain ownership

Unified operations should not erase specialist responsibility.

Network engineers still need network depth. Storage teams need storage expertise. Hardware teams need component diagnostics. Application teams need transaction and code level insight.

The common platform should support collaboration by showing:

Shared incident
Affected services
Relevant infrastructure
Current owner
Contributing events
Recent changes
Diagnostic links
Actions taken

Each team can then use specialist tools when deeper analysis is required.

The unified layer coordinates the response. It does not replace every expert console.

Link infrastructure to business services

Technology teams naturally think in domains.

Users experience services.

A business service may depend on:

Load balancer
Web servers
Application servers
Database
Storage
Network
Identity provider
Cloud API
Power
Cooling

A failure in any of these areas can produce the same user complaint.

Without service relationships, each team sees only its own part.

Business service context helps answer:

Which service is affected?
How critical is it?
Which users are affected?
Which infrastructure supports it?
Is redundancy available?
What changed recently?
Who should lead the incident?
Which recovery action has the lowest risk?

The business service intelligence guide describes how infrastructure and operational data can be connected to business impact.

Use one incident workflow across domains

Separate tools often create separate incident processes.

One team uses email. Another uses chat. Another creates service desk tickets. A vendor portal contains additional updates.

This creates fragmented records.

A shared workflow should define:

Incident creation
Severity
Ownership
Escalation
Collaboration
Change approval
Resolution
Closure
Review

Supporting data can remain in specialist tools, but the operational record should be centralized.

A shared incident should include:

Timeline
Alerts
Affected assets
Service impact
Recent changes
Actions
Decisions
Owners
Vendor cases
Resolution evidence

This reduces repeated communication and improves post incident learning.

Standardize integration patterns

Point to point integrations can become another source of complexity.

If every tool connects directly to every other tool, the number of relationships grows quickly.

A more sustainable approach uses standard integration patterns.

Examples include:

Event ingestion
Asset synchronization
Ticket creation
Workflow trigger
Data export
API query
Message bus
Webhook
Scheduled reconciliation

Integration standards should define:

Authentication
Data format
Field mapping
Retry behavior
Failure monitoring
Rate limits
Ownership
Versioning
Audit logging

This makes it easier to add or replace vendors without rebuilding the entire operational architecture.

Avoid copying every metric into one platform

Centralizing every raw metric can create cost and complexity.

Problems may include:

Large storage requirements
Slow queries
Duplicate data
Expensive licensing
Difficult upgrades
Unclear ownership
Reduced specialist functionality

The unified layer should collect the data needed for shared decisions.

This may include:

Health
Availability
Capacity
Key performance indicators
Alerts
Configuration
Relationships
Changes
Ownership

Deep historical telemetry can remain in domain systems when appropriate.

The architecture should support drill through from shared context to specialist data.

Keep source links and provenance visible

When data is aggregated, users need to know where it came from.

A useful record should show:

Source system
Collection time
Original event
Normalized event
Last update
Confidence
Related object
Field ownership

This helps operators decide whether data is current and reliable.

It also makes troubleshooting integrations easier.

If the unified platform shows a storage volume as healthy while the vendor tool shows a warning, provenance allows the team to identify whether the issue is stale data, mapping, collection failure, or interpretation.

Treat multi vendor compatibility as a lifecycle responsibility

Compatibility is not a one time project.

It changes when:

New device models are purchased
Firmware is upgraded
APIs change
Protocols are deprecated
Vendors are acquired
Cloud services evolve
Security settings change
Certificates expire
Authentication requirements change

The operating model should include:

Compatibility testing
Connector ownership
Upgrade review
Regression testing
Vendor communication
Failure monitoring
Fallback methods
Documentation

A platform that supports a device today may require updates to continue collecting useful data tomorrow.

Use automation carefully across vendors

Multi vendor control is harder than multi vendor monitoring.

A standardized action such as reboot may have different:

APIs
Permissions
Safety requirements
Response behavior
Audit requirements
Failure modes

Automation should distinguish between:

Read only collection
Diagnostic action
Configuration change
Power control
Firmware update
Workload movement

Higher risk actions need stronger controls.

These may include:

Approval
Role restrictions
Maintenance windows
Prechecks
Rollback
Recording
Confirmation
Post action validation

The goal is consistent governance, even when the underlying vendor commands differ.

Reduce tool overlap using evidence

Once data and workflows are unified, the organization can evaluate tool overlap.

Questions include:

Which tools collect the same data?
Which dashboards are still used?
Which integrations are duplicated?
Which alerts are unique?
Which specialist capabilities are essential?
Which tools have no clear owner?
Which tools create manual work?
Which contracts are approaching renewal?

Tools can then be classified as:

Strategic
Specialist
Temporary
Redundant
Retireable

This is more reliable than selecting a target number of tools before understanding their operational value.

The guide to unified IT infrastructure operations provides a broader framework for combining visibility, workflows, and operational control across infrastructure domains.

Organize teams around shared services and domain expertise

Technology silos are not created only by software.

They are also created by team structures, incentives, and communication.

A practical model combines:

Domain ownership
Shared incident process
Service ownership
Common data standards
Cross domain review
Joint capacity planning
Shared reliability objectives

For example, a storage team may own storage engineering while participating in service reviews for the applications that depend on it.

This keeps specialist expertise while improving shared accountability.

Measure whether silos are actually decreasing

Useful measures include:

Number of monitoring tools
Number of duplicate alerts
Time spent switching tools
Mean time to identify the responsible domain
Mean time to resolve cross domain incidents
Percentage of assets with clear ownership
Percentage of alerts linked to services
Percentage of incidents with one shared record
Number of manual integrations
Data reconciliation errors
User adoption of the unified platform
Tool retirement savings

The goal is not to maximize data centralization.

The goal is to improve operational outcomes.

A practical implementation sequence

A phased approach reduces risk.

Phase 1: Inventory the current landscape

Document:

Tools
Vendors
Data sources
Integrations
Owners
Costs
Users
Workflows
Gaps

Phase 2: Define the common model

Agree on:

Objects
Identifiers
Ownership
Relationships
Severity
Lifecycle states
Service context

Phase 3: Unify events

Implement:

Alert ingestion
Normalization
Deduplication
Correlation
Routing
Shared incidents

Phase 4: Reconcile assets

Connect:

Discovery
CMDB
DCIM
Cloud inventory
Procurement
Lifecycle data

Phase 5: Add service context

Map:

Applications
Dependencies
Owners
Criticality
Business impact

Phase 6: Standardize workflows

Connect:

Incident
Change
Maintenance
Capacity
Automation
Reporting

Phase 7: Rationalize tools

Retire only after:

Data is preserved
Workflows are replaced
Users are trained
Integrations are validated
Specialist depth remains available

Unification should reduce friction, not erase complexity

Multi vendor infrastructure will remain complex.

Different technologies have different operating models, failure modes, and diagnostic requirements.

A useful unified platform does not pretend those differences do not exist.

It provides a shared structure for:

Identity
Health
Ownership
Relationships
Events
Incidents
Changes
Services
Decisions

Specialist tools remain available when teams need deeper control.

The result is not one tool that replaces everything.

It is one operating model that helps many tools, teams, and vendors work as one service organization.

That is how enterprises can manage multi vendor infrastructure without creating another silo.

Originally published on the Sensaka blog.

Why IT Asset Registers Stop Matching the Real Infrastructure

Da — Sat, 01 Aug 2026 08:02:00 +0000

Most IT asset registers begin with good intentions.

A team creates a spreadsheet, imports procurement records, builds a configuration management database, or deploys an asset management platform. Devices are assigned owners, locations, serial numbers, warranty dates, and business purposes.

For a period of time, the records appear reliable.

Then the infrastructure changes.

Servers are moved. Components are replaced. Virtual machines are created. Network ports are reassigned. Storage capacity is expanded. Equipment is transferred between sites. Hardware is retired but remains powered on. New devices enter production before records are completed.

The asset register slowly stops matching reality.

This problem is often treated as a data entry issue. In practice, it is usually an operating model issue. The infrastructure changes continuously, while the asset process depends on periodic updates, manual handoffs, and disconnected systems.

A trustworthy asset register must be designed around change.

Asset data begins aging as soon as it is created

An asset record is a statement about the infrastructure at a particular moment.

It may describe:

Device identity
Serial number
Manufacturer
Model
Configuration
Physical location
Rack and U position
Owner
Business service
Warranty
Support contract
Lifecycle status
Network address
Power connection
Software relationship

The moment any of these details change, the record begins to drift.

A server can remain in the same rack while its memory, disks, network adapters, firmware, operating role, and owner all change. A device may be physically removed but remain active in monitoring. A replacement unit may inherit the same hostname while having a different serial number.

Asset accuracy therefore requires more than an initial inventory.

It requires a continuous way to detect, verify, and record change.

The guide to building an IT asset source of truth explains why trustworthy records depend on clear ownership, automated evidence, and controlled synchronization.

Procurement data is necessary but incomplete

Procurement systems are often the starting point for asset records.

They usually contain useful information such as:

Purchase order
Supplier
Cost
Contract
Delivery date
Warranty
Product description
Quantity
Cost center

This information is valuable, but it does not prove what is currently installed.

Several differences may appear between purchase and production:

Equipment is delivered in batches
Devices are assigned to different sites
Components are substituted
Memory or storage is expanded
Spare units remain in storage
Equipment is returned
Devices are rebuilt for another purpose
Serial numbers are entered incorrectly
Bundled items are recorded as one line
Contract descriptions do not match technical configurations

Procurement records explain what the organization intended to buy.

Operations data must explain what is actually present and how it is being used.

Both sources are needed, but they answer different questions.

Manual inventory creates a temporary snapshot

Physical inventory exercises can improve asset accuracy.

Teams may scan labels, inspect racks, compare serial numbers, and update spreadsheets. This can reveal missing, moved, or undocumented equipment.

The limitation is that manual inventory creates a snapshot.

The data begins changing again immediately after the exercise.

Manual inventory also has practical weaknesses:

Labels may be inaccessible
Serial numbers may be difficult to read
Components inside a device are not visible
Equipment may be powered off
Devices may be temporarily moved
Hostnames may not match physical labels
Staff may interpret fields differently
Large sites require significant labor
Results may take weeks to reconcile
Business relationships are rarely visible from the rack

Manual checks remain useful for verification, but they cannot be the only mechanism for maintaining a live asset register.

The infrastructure changes through many separate processes

Asset drift occurs because change enters through many paths.

Examples include:

Procurement
Project deployment
Emergency replacement
Maintenance
Incident response
Capacity expansion
Data center migration
Cloud provisioning
Virtualization
Network change
Security remediation
Business application rollout
Decommissioning
Vendor support
Test environment activity

Each process may use different tools, teams, and approval paths.

A hardware engineer may replace a disk during an incident. The service desk records the incident, but the asset platform does not record the component change.

A project team may deploy new servers. The devices enter monitoring, but the ownership and warranty fields remain blank.

A facilities team may move equipment between racks. The rack diagram is updated, but the CMDB still shows the previous location.

The problem is not that people do not care about data quality. The problem is that asset updates are often separate from the work that changes the assets.

Device identity is harder than it appears

A reliable register must determine whether two records represent:

The same device
Different devices
A replacement
A duplicate
A virtual object
A physical component
A renamed asset
A rebuilt asset

Common identifiers include:

Serial number
Asset tag
Hostname
IP address
MAC address
Management controller address
Cloud instance ID
Chassis ID
Virtual machine ID
Rack position

Each identifier has limitations.

Hostnames can change. IP addresses can be reassigned. Asset labels can be missing. Serial numbers can be entered inconsistently. Virtual machine IDs may change during migration. Chassis and blades may have separate identities.

A strong asset model uses several identifiers and defined matching rules.

For example, a server might be matched through a combination of:

Manufacturer
Model
Serial number
Management controller
Network interface
Physical location

This reduces the chance that the same device appears several times under different names.

Component changes are often invisible to the register

Many asset systems track equipment at the device level.

That is useful, but important changes occur inside the device.

Examples include:

Memory replacement
Disk replacement
CPU change
Network adapter change
Power supply replacement
RAID controller replacement
Firmware update
Battery replacement
Storage expansion
Optical module replacement

These changes can affect:

Performance
Capacity
Warranty
Compatibility
Compliance
Failure risk
Expansion options
Support eligibility

A server can still have the same hostname and serial number while its internal configuration no longer matches the approved baseline.

Component level collection helps the organization understand what is actually installed rather than what the original purchase record described.

The hardware lifecycle management guide describes how configuration, warranty, maintenance, replacement, and retirement data should remain connected throughout the life of the equipment.

The CMDB and the asset register are related but different

Organizations often use the terms asset register and CMDB as though they mean the same thing.

They overlap, but their purposes are different.

An asset register commonly focuses on:

Ownership
Cost
Contract
Warranty
Lifecycle
Physical identity
Financial control

A CMDB commonly focuses on:

Configuration items
Relationships
Services
Dependencies
Changes
Incidents
Operational impact

One system may support both functions, but the data model and ownership still need to be clear.

For example:

Procurement may own purchase cost
Asset management may own lifecycle status
Infrastructure discovery may own technical configuration
DCIM may own physical location and power relationships
Network systems may own port and IP data
CMDB may own service relationships
ITSM may own incidents and changes

The overview of what a CMDB is can help teams distinguish configuration management from financial and physical asset control.

The goal is not to force every field into one platform. The goal is to define which source is trusted for each field and how changes move between systems.

Duplicate records reduce confidence quickly

Duplicate assets appear for many reasons:

Different discovery tools create separate records
A hostname changes
A device receives a new IP address
A serial number contains formatting differences
One system records the chassis while another records the blade
A replacement device inherits an old name
Data is imported more than once
Records are created before discovery
Cloud resources are recreated automatically

Duplicates create several operational problems.

They can cause:

Incorrect device counts
Duplicate alerts
Wrong ownership
Conflicting lifecycle status
Inaccurate capacity reports
Incorrect maintenance coverage
Confusion during incidents
Failed reconciliation

Removing duplicates manually can help, but the problem returns unless matching and reconciliation rules improve.

A strong process should define:

Which identifiers are authoritative
How records are matched
How conflicts are resolved
Which system can create new records
Which system can retire records
How uncertain matches are reviewed

Missing assets are often more dangerous than inaccurate fields

An incorrect warranty date is a problem.

An entirely unknown device can be a larger risk.

Unrecorded assets may include:

Emergency replacements
Test servers
Temporary network devices
Vendor appliances
Spare equipment connected to power
Devices added during projects
Equipment inherited through acquisition
Unsupported legacy systems
Shadow IT
Cloud resources created outside standard processes

These assets may not receive:

Monitoring
Patching
Security review
Warranty management
Backup
Ownership
Capacity planning
Decommissioning

Discovery should therefore look for both inaccurate records and infrastructure that has no record at all.

Retired equipment often remains in the data

Asset registers commonly contain devices that no longer exist in production.

Records remain because:

Decommissioning was not completed
Disposal evidence is missing
Monitoring was disabled but the asset was not retired
Equipment was moved to storage
The system requires fields that no one can confirm
The owner left the organization
The asset is involved in an unresolved financial process
Old records are retained for audit without clear status

This creates confusion between:

Active
Inactive
In storage
Reserved
Under repair
Retired
Disposed
Lost

Lifecycle states should be explicit.

Historical records should remain available for audit, but they should not appear as active production capacity.

Cloud and virtualization increase the rate of change

Physical equipment may remain in place for years.

Virtual and cloud resources can appear and disappear within minutes.

This creates a different asset management challenge.

Cloud and virtual objects may include:

Instances
Virtual machines
Containers
Volumes
Images
Load balancers
Network interfaces
Databases
Clusters
Serverless functions

These resources may be created automatically by:

Deployment pipelines
Autoscaling
Infrastructure as code
Test automation
Development teams
Managed services

Manual registration cannot keep pace.

Cloud asset accuracy depends on API based discovery, tagging standards, ownership rules, and lifecycle automation.

The physical and cloud models should also be connected where business services depend on both.

Ownership fields decay when organizations change

Even when technical data remains accurate, ownership data can become stale.

People move roles. Teams are reorganized. Applications are transferred. Vendors change. Support responsibilities are outsourced.

A record may still name:

A former employee
A dissolved team
An expired project
An old cost center
A vendor that no longer provides support

Ownership should be tied to durable organizational structures where possible.

For example:

Service owner
Support group
Business unit
Cost center
Technical domain
Location team

Individual contacts can still be included, but the record should not depend entirely on one person.

Periodic ownership attestation can help identify records that no longer have a responsible team.

Warranty data becomes unreliable without serial level verification

Warranty and maintenance records often come from contracts or supplier lists.

Problems arise when:

Serial numbers are missing
Equipment was replaced
Coverage differs by component
Support was renewed for only part of the estate
Devices moved between regions
Contracts use different asset identifiers
Supplier records do not match internal records

A trustworthy warranty process should connect:

Device identity
Component identity
Contract
Coverage period
Support level
Supplier
Location
Lifecycle status

This allows teams to answer practical questions before a failure occurs:

Is the device covered?
Which supplier should be contacted?
What is the response time?
Is onsite service included?
Are replacement parts available?
Will support expire soon?

Asset data quality should be measured by field and purpose

A single percentage for asset accuracy can hide important weaknesses.

Different fields have different operational value.

Examples include:

Serial number accuracy
Location accuracy
Ownership completeness
Warranty completeness
Configuration accuracy
Relationship accuracy
Lifecycle status accuracy
Last verification date
Discovery coverage
Duplicate rate

The required level of accuracy depends on the use case.

For financial reporting, purchase cost and depreciation may be critical.

For incident response, hostname, location, hardware health, owner, and service relationship may matter more.

For capacity planning, rack position, power, space, and network connections are essential.

Data quality should therefore be measured against the decisions the data must support.

Automation reduces drift but does not eliminate governance

Automated discovery can collect:

Manufacturer
Model
Serial number
CPU
Memory
Storage
Network interfaces
Firmware
Power state
Hardware health
IP address
Virtual machine data
Cloud metadata

This reduces manual effort and increases update frequency.

However, automation cannot always determine:

Business owner
Financial treatment
Intended use
Criticality
Contract responsibility
Disposal approval
Data classification

These fields still require governance and workflow.

The best model combines automated technical evidence with controlled business data.

Reconciliation should be a continuous process

Reconciliation compares sources and decides which value should be trusted.

A continuous process may include:

Discover assets automatically.
Match them to existing records.
Identify missing, duplicate, and conflicting records.
Apply field ownership rules.
Route unresolved conflicts to the right team.
Record the source and verification time.
Update dependent systems.
Track recurring error patterns.

This is more effective than occasional large cleanup projects.

The purpose is not to make every system identical. It is to keep shared facts consistent and explain differences when they are intentional.

Select tools based on the operating model

Asset tools vary in focus.

Some emphasize:

Financial asset management
Software licensing
Endpoint inventory
Service configuration
Physical data center assets
Cloud resources
Network discovery
Hardware lifecycle
Procurement and contracts

The best choice depends on what the organization needs to control.

The report on IT asset management software provides a framework for comparing products by scope, discovery, lifecycle support, integrations, reporting, and operational fit.

The tool should support the data ownership model rather than forcing all processes into one generic structure.

Build asset updates into daily work

Asset accuracy improves when updates occur naturally inside operational processes.

Examples include:

Procurement creates a planned asset record
Delivery confirms serial numbers
Installation records location and connections
Discovery validates configuration
Monitoring confirms active status
Change management records modifications
Incident workflows record replacements
Maintenance updates warranty and service history
Decommissioning changes lifecycle status
Disposal records final evidence

This design reduces the need for separate data entry.

The asset register becomes a result of operational work rather than an additional administrative task.

A trustworthy register needs visible evidence

Users trust asset data when they can see:

Where a value came from
When it was last verified
Which system owns it
Who changed it
What the previous value was
Whether the value was discovered or entered manually
Whether sources disagree

Evidence makes uncertainty manageable.

For example:

Serial number discovered from the management controller
Rack position confirmed by physical audit
Owner confirmed by service team
Warranty imported from supplier
Configuration last collected two hours ago

This is more useful than presenting every field as equally certain.

The goal is decision quality

An asset register is valuable only when it improves decisions.

A trustworthy register helps teams:

Find equipment quickly
Understand ownership
Plan capacity
Respond to failures
Verify warranty
Manage change
Support audits
Reduce duplicate purchases
Retire unused assets
Understand service dependencies
Improve security coverage
Forecast replacement

When the data is unreliable, teams create side spreadsheets and personal lists.

That behavior is a warning. It shows that the official system is no longer trusted.

Asset accuracy is an ongoing capability

IT asset registers stop matching reality because infrastructure changes continuously while record keeping is often periodic and manual.

The solution is not one large cleanup.

It is a continuous operating capability built from:

Automated discovery
Stable identity rules
Clear field ownership
Continuous reconciliation
Lifecycle workflows
Change integration
Evidence
Data quality measurement
User accountability

The most reliable asset register is the one that learns about changes as they happen.

When technical evidence and business governance work together, the register becomes more than an inventory. It becomes a trusted foundation for operations, finance, risk, and planning.

Originally published on the Sensaka blog.

How to Improve PUE Without Compromising Reliability

Da — Sat, 01 Aug 2026 07:56:53 +0000

Power Usage Effectiveness is one of the most widely used data center efficiency metrics.

It compares total facility energy with the energy consumed by IT equipment. The closer the result is to 1.0, the smaller the share of energy used by cooling, power conversion, lighting, and other supporting systems.

This makes PUE useful, but it can also create the wrong behavior.

A team that pursues a lower number without considering resilience may reduce safety margins, disable redundancy, raise temperatures too aggressively, or make changes that improve the metric while increasing operational risk.

The objective should not be the lowest possible PUE at any cost.

The objective should be to reduce avoidable facility energy while continuing to protect equipment, workloads, and business services.

Start with a consistent measurement boundary

PUE improvement begins with reliable measurement.

The basic relationship is:

Total facility energy divided by IT equipment energy.

The calculation appears simple, but results can vary according to where and how energy is measured.

Questions to define include:

Which facility loads are included?
Where is total energy measured?
Where is IT energy measured?
Are shared building systems included?
Are office areas excluded?
Is generator fuel included?
Are cooling pumps included?
Are lighting and security included?
Is measurement continuous or based on samples?
Are annual and peak values reported separately?

If the boundary changes, the result may improve without any real efficiency gain.

For example, moving a cooling load outside the measurement boundary does not reduce energy consumption. It only changes the accounting.

The guide to what PUE is provides the core definition and explains the main factors that influence the metric.

Use annual PUE, not only a single snapshot

PUE changes with:

Weather
IT load
Cooling mode
Equipment utilization
Time of day
Maintenance activity
Seasonal conditions
Facility occupancy

A value measured during a cool night at high IT load may look excellent. The same facility may perform very differently during a hot afternoon or at low utilization.

Useful reporting should include:

Annual PUE
Monthly PUE
Daily trend
Peak conditions
Low load conditions
Seasonal variation
Load level
Weather context
Planned maintenance periods
Meter quality

An interactive PUE calculator can help teams understand the relationship between facility energy and IT energy, but operational improvement depends on continuous, consistent data.

Improve the denominator carefully

PUE can improve when IT load increases, even if total energy also increases.

This happens because fixed facility loads are spread across more IT consumption.

For example, lighting, security, controls, and some cooling systems may consume a similar amount of energy whether the facility is lightly or heavily loaded.

This means a lower PUE can result from:

Better facility efficiency
Higher IT utilization
More IT equipment
Changes in measurement
Seasonal weather

These are not equivalent.

Teams should therefore report PUE alongside:

Total facility energy
IT energy
IT utilization
Rack utilization
Workload output
Cooling energy
Power losses
Electricity cost

A lower PUE should reflect real efficiency improvement rather than simply a larger denominator.

Airflow management is often the safest first step

Many facilities cool more aggressively than necessary because cold air and hot air are not controlled effectively.

Common airflow problems include:

Missing blanking panels
Unsealed cable openings
Open rack spaces
Poor containment
Obstructed floor tiles
Incorrect perforated tile placement
Hot air recirculation
Cold air bypass
Uneven rack density
Poor equipment orientation
Excess airflow

These problems cause the cooling system to work harder while some equipment still receives air at the wrong temperature.

Low risk improvements include:

Install blanking panels
Seal cable openings
Remove airflow obstructions
Align equipment intake and exhaust direction
Balance perforated tile placement
Close unused rack openings
Separate hot and cold air
Adjust fan speeds using measured demand
Review rack layouts
Remove abandoned cabling that blocks airflow

Airflow improvement can reduce cooling energy without changing equipment temperature limits or redundancy.

Containment should be designed around failure behavior

Hot aisle or cold aisle containment reduces mixing between supply and return air.

This can improve:

Cooling efficiency
Temperature consistency
Cooling capacity
Rack density
Predictability

However, containment changes how the room behaves during:

Cooling failure
Fan failure
Door opening
Power interruption
Fire events
Maintenance
Sensor failure

Before deployment, evaluate:

Emergency airflow
Pressure balance
Fire suppression
Door control
Leak paths
Human access
Temperature rise rate
Sensor placement
Backup cooling
Failure alarms

Containment should improve normal efficiency while preserving a safe response during abnormal conditions.

The guide to data center cooling systems outlines the major cooling approaches and the operational considerations behind them.

Raise temperature setpoints based on evidence

Increasing supply air temperature can reduce cooling energy.

It may enable:

More economizer hours
Higher chiller efficiency
Lower compressor load
Reduced fan energy
Better use of ambient conditions

However, setpoint changes should be based on equipment inlet temperature, not only room temperature.

A safe process includes:

Verify sensor accuracy.
Place sensors at representative equipment inlets.
Identify the hottest racks.
Review vendor temperature limits.
Check redundancy and cooling capacity.
Increase setpoints in small steps.
Observe equipment temperatures and alarms.
Test during high load and hot weather.
Document rollback criteria.
Continue trend monitoring.

A room level sensor may show an acceptable value while upper rack equipment receives much warmer air.

The goal is to avoid overcooling while maintaining acceptable inlet conditions for every critical device.

Use variable speed equipment where possible

Fans and pumps often consume less energy when speed is reduced.

Many cooling systems can adjust output according to actual demand through:

Variable frequency drives
Electronically commutated fans
Variable speed pumps
Dynamic pressure control
Temperature based control
Load based control

This can reduce energy compared with running equipment continuously at full speed.

Control changes must be tested carefully.

Risks include:

Slow response to rapid load changes
Incorrect sensor input
Poorly tuned control loops
Minimum flow limits
Uneven pressure
Sensor failure
Communication failure

Safe optimization requires monitoring, alarms, minimum operating limits, and fallback behavior.

Match cooling output to the real heat load

Overcooling often occurs when cooling systems operate according to static assumptions.

A better approach uses real time information such as:

Rack power
Inlet temperature
Return temperature
Air pressure
Humidity
Cooling unit load
IT workload
Weather
Chilled water temperature
Flow rate

This data can help operators reduce unnecessary cooling while maintaining thermal margin.

The most effective control point depends on the facility design.

For example:

Room based cooling may use zone temperature and pressure
In row cooling may respond to local rack load
Rear door heat exchangers may respond to water and exhaust temperature
Liquid cooling may respond to flow, pressure, and coolant temperature

Automation should remain transparent. Operators need to understand what changed, why it changed, and how to return to a safe state.

Expand economizer use when conditions allow

Air side and water side economizers use favorable outdoor conditions to reduce mechanical cooling.

Potential benefits include:

Lower compressor energy
Reduced chiller operation
Lower cooling cost
Improved seasonal PUE

Practical constraints include:

Climate
Air quality
Humidity
Contamination
Water availability
Control complexity
Maintenance
Local regulations

Economizer performance should be evaluated across the full year.

A design that works well in a cool, dry climate may be unsuitable in a hot, humid, polluted, or water constrained location.

Reliability controls should include:

Environmental monitoring
Filtration
Humidity control
Automatic mode switching
Alarm thresholds
Backup mechanical cooling
Maintenance procedures

Reduce power conversion losses

Energy is lost as power moves through the facility.

Losses may occur in:

Transformers
UPS systems
Batteries
Power distribution units
Cables
Power supplies
Voltage conversion stages

Efficiency improvements can include:

Operate UPS systems within efficient load ranges
Consolidate lightly loaded UPS modules
Use efficient transformer designs
Reduce unnecessary conversion stages
Improve voltage matching
Balance phases
Maintain clean electrical connections
Replace inefficient legacy equipment
Review distribution topology

Redundancy requirements must remain intact.

For example, consolidating UPS modules may improve efficiency, but the final configuration must still support the agreed failure and maintenance scenarios.

Efficiency cannot be assessed only during normal operation. It must also be assessed during component failure, bypass, testing, and maintenance.

Eliminate ghost load and abandoned equipment

Unused or underused equipment still consumes power and creates heat.

Examples include:

Powered but idle servers
Abandoned network equipment
Old storage systems
Duplicate appliances
Unused development systems
Spare devices left online
Legacy equipment after migration
Empty chassis with active components

Removing this equipment can reduce:

IT energy
Cooling load
Rack usage
Port usage
Maintenance effort
Monitoring noise

However, there is an important PUE effect.

Removing IT load can sometimes make PUE appear worse because the denominator becomes smaller while fixed facility loads remain.

This does not mean the action was harmful.

Total energy and operating cost may still decline significantly.

That is why PUE should never be used alone.

Improve IT utilization as well as facility efficiency

A facility can have an excellent PUE while supporting poorly utilized servers.

IT efficiency improvements may include:

Consolidate workloads
Retire unused systems
Improve virtualization
Use power management
Schedule noncritical workloads
Match hardware to workload
Refresh inefficient equipment
Improve storage efficiency
Reduce duplicate environments
Use autoscaling where appropriate

These measures reduce energy per useful unit of computing.

They may not always improve PUE because they reduce IT energy, but they can improve the overall environmental and financial outcome.

A balanced program should track both facility efficiency and IT productivity.

Use device level power data for placement decisions

Rack placement affects cooling and power efficiency.

If high density equipment is concentrated without planning, the facility may need excessive airflow or lower temperature setpoints to protect one hotspot.

Better placement considers:

Actual device power
Rack power capacity
Cooling capacity
Airflow
Redundant feeds
Weight
Network requirements
Future growth
Maintenance access

Spreading load evenly may help some facilities. Concentrating high density equipment into purpose built zones may help others.

The correct strategy depends on cooling architecture and distribution design.

A data center power calculator can support initial rack estimates, while real device telemetry should guide ongoing decisions.

Optimize humidity control without creating instability

Legacy humidity control strategies can consume significant energy.

Facilities may humidify and dehumidify at the same time if systems are poorly coordinated.

Improvement opportunities include:

Review acceptable humidity ranges
Calibrate sensors
Coordinate cooling units
Reduce overlapping control
Use dew point based control where appropriate
Improve control deadbands
Review seasonal behavior
Eliminate unnecessary humidification

Changes should protect equipment from:

Condensation
Static risk
Corrosion
Rapid environmental change

Humidity control should be optimized carefully and verified across the facility, not adjusted based on one sensor.

Maintain equipment to preserve efficiency

Dirty, worn, or poorly calibrated equipment consumes more energy.

Maintenance actions that support efficiency include:

Clean filters
Clean coils
Inspect heat exchangers
Calibrate sensors
Repair leaking valves
Maintain pumps
Check fan performance
Verify refrigerant levels
Inspect dampers
Maintain cooling towers
Test controls
Review alarm history

Maintenance also protects reliability.

Efficiency projects often focus on new technology, but restoring existing systems to proper condition may deliver faster and safer results.

Avoid disabling redundancy for a better number

Some efficiency measures can reduce redundancy.

Examples include:

Turning off redundant cooling units
Reducing active UPS modules
Closing backup airflow paths
Raising setpoints without thermal margin
Reducing pump capacity
Operating near circuit limits
Reducing spare capacity

These measures may lower PUE during normal operation, but they can increase risk during:

Equipment failure
Maintenance
Utility disturbance
Sudden workload increase
Extreme weather
Control failure

A safe optimization process should define the minimum required resilience.

This may include:

N plus one cooling
Dual power paths
UPS reserve
Generator reserve
Temperature recovery time
Spare capacity
Maintenance tolerance
Failure response

The efficiency target must operate inside these limits.

Test changes under realistic conditions

A change that works during low load may fail during peak demand.

Testing should include:

Peak IT load
Hot weather
Low facility load
One cooling unit unavailable
One power path unavailable
Maintenance mode
Rapid workload change
Sensor failure
Communication failure
Emergency response

Not every test requires a live failure. Modeling, staged tests, and controlled maintenance windows can provide useful evidence.

The important point is that optimization should be validated beyond ideal conditions.

Use change control for energy optimization

Efficiency changes should follow the same discipline as other production changes.

A strong process includes:

Baseline measurement
Defined objective
Risk review
Affected equipment list
Success criteria
Alarm thresholds
Rollback plan
Change window
Monitoring period
Final review

For example, a temperature setpoint increase should specify:

Current temperature
Proposed temperature
Expected energy effect
Maximum equipment inlet temperature
Alarm threshold
Test duration
Rollback condition
Responsible owner

This turns efficiency improvement into a controlled operational program.

Track several metrics together

PUE is more useful when combined with other measures.

Recommended measures include:

Total facility energy
IT energy
Cooling energy
Electricity cost
Peak demand
Rack power
Server utilization
Temperature compliance
Number of thermal alarms
Cooling reserve
UPS efficiency
Water use
Carbon intensity
Workload output
Service availability

A change should be considered successful when it reduces waste without causing:

More alarms
Higher equipment temperature risk
Reduced redundancy
More downtime
More maintenance
Unstable controls
Lower workload performance

Prioritize improvements by risk and payback

Not every facility needs the same sequence.

A practical order often begins with:

Correct measurement.
Remove airflow problems.
Calibrate sensors.
Repair inefficient equipment.
Optimize setpoints gradually.
Tune fan and pump control.
Improve containment.
Remove unused IT equipment.
Improve placement and utilization.
Evaluate major cooling upgrades.

This order starts with relatively low risk measures before moving toward larger capital projects.

Each action should be evaluated for:

Energy reduction
Cost
Implementation effort
Operational risk
Payback period
Reliability effect
Maintenance effect

PUE improvement is a continuous operating practice

There is no permanent final PUE.

The number changes as:

Workloads grow
Equipment is refreshed
Rack density changes
Weather changes
Cooling systems age
Sensors drift
Facilities expand
Operating practices change

Teams should review efficiency regularly and connect it to capacity, maintenance, and lifecycle planning.

The most effective programs combine:

Accurate measurement
Facility telemetry
IT power data
Environmental monitoring
Capacity data
Change management
Maintenance
Business priorities

Reliability is the boundary condition

PUE is valuable because it exposes facility overhead.

It helps teams find waste in cooling, power conversion, airflow, and supporting systems.

But a data center exists to provide reliable computing.

An efficiency change that increases outage risk, thermal instability, or maintenance exposure can create costs far greater than the energy saved.

The right target is therefore not the lowest theoretical PUE.

It is the lowest sustainable PUE that the facility can achieve while maintaining its agreed resilience, environmental limits, and service obligations.

That balance produces real efficiency rather than a better number alone.

Originally published on the Sensaka blog.

DEV Community: Da

"A Pod at 40 Percent CPU Can Still Freeze Every 100 Milliseconds"

A Pod at 40 Percent CPU Can Still Freeze Every 100 Milliseconds

CPU percentages hide the time dimension

CFS quota is enforced in short periods

Low average usage and high throttling are not contradictory

The averaging trap is worst for latency-sensitive services

Requests and limits solve different problems

Why some operators remove CPU limits

Why some operators keep CPU limits

For latency-sensitive services, load testing is the deciding tool

The number of threads changes how quickly quota disappears

kubectl top is not enough

PSI detects pressure that quota metrics miss

Hypervisor steal time is another invisible failure mode

QoS class does not eliminate bad CPU sizing

CPU limits do not protect a service from bad requests

Accurate requests are harder than they look

Queueing theory explains why predictability costs money

A practical troubleshooting path starts below the dashboard

The right CPU policy is workload-specific

FAQ

Can a Pod be throttled while showing only 40 percent CPU?

What causes CFS CPU throttling?

Why do multiple threads matter?

Which metrics show actual throttling?

What does CPU pressure show?

Should latency-sensitive Pods have CPU limits?

When are CPU limits most useful?

Is a realistic CPU request still important without a limit?

CTA

"Cl0p Targets PTC Windchill: Why Product Lifecycle Systems Need Better Backup Planning"

A Patch Stops a Vulnerability but Does Not Erase an Intrusion

Windchill Recovery Requires More Than Application Files

Clean Recovery Points Need Independent Protection

Recovery Testing Should Include a Compromise Scenario

"HollowFrame and Matryoshka Malware Target Law Firms Through Spear Phishing"

The Attack Separated Malicious Activity Across Stages

Confidentiality Risk Extends Beyond Privileged Documents

Law Firms Need Controls Around Normal Document Work

Vendor and Workflow Reviews Must Include Security

"Boston Law Firm Raises Starting Salaries to $235,000: What It Means for BigLaw Competition"

Compensation Is Becoming a Market Signal

Higher Salaries Increase the Cost of Time Leakage

Productivity Cannot Mean Unreviewed Billing

The Competition Will Extend Beyond Base Salary

"VMware to Proxmox Migration: Lessons From Centrogest’s Three Node Cluster"

The Migration Solved a Storage Problem First

Backup Was Built as a Separate Recovery Layer

Power and Operations Belong in the Migration Plan

Why AI Data Centers Need More Than GPU Monitoring

GPU monitoring answers a narrow set of questions

AI workloads span multiple infrastructure domains

Workload context explains who is using the GPU

Network monitoring explains communication waits

Storage monitoring explains data starvation

Physical hardware monitoring finds faults above and below the operating system

Cooling monitoring protects performance and capacity

Usage metering explains efficiency and accountability

Observability requires correlation, not more isolated dashboards

AI data center operations also need action and accountability

A broader operating model for AI infrastructure

Resource visibility

Workload visibility

Relationship visibility

Performance visibility

Operational visibility

The shift from monitoring accelerators to operating AI infrastructure

The Hidden Network and Storage Bottlenecks Behind Slow AI Training

Slow training is a system problem

Distributed training amplifies small network problems

Packet loss and retransmission consume training time

Network topology affects communication efficiency

Storage can starve otherwise healthy GPUs

Throughput alone does not explain storage performance

Checkpointing creates another storage pressure point

Data preprocessing can look like a storage problem

Shared infrastructure creates hidden contention

Average metrics hide stragglers

Correlation is more useful than dashboard switching

`kubectl top` is not enough