Introduction
Imagine submitting an MPI job to an HPC cluster. It starts normally, but something does not feel right.
A job that usually finishes in 20 minutes is now taking 45 minutes. Some MPI ranks appear to be waiting for others. Another job occasionally fails with communication errors. You check CPU and memory utilisation, but nothing looks unusual.
Could the problem be the InfiniBand network?
InfiniBand is designed to provide high bandwidth and low latency communication between compute nodes. It is widely used in HPC environments because applications such as MPI depend on fast communication between nodes.
However, even a high performance network can experience problems. A damaged cable, a faulty transceiver, a problematic switch port or other physical and network issues can affect communication.
The difficult part is that these problems are not always obvious. A node might remain online, network interfaces might show an active link, and jobs might still run. Yet performance can be significantly affected.
This is where InfiniBand diagnostic tools become useful.
In this article, we will look at one particularly useful tool called ibqueryerrors. We will understand what it does, what different errors mean, what symptoms users might experience and how an HPC administrator can investigate potential network problems.
First, what is InfiniBand?
Before looking at the troubleshooting commands, let's understand what we are troubleshooting.
In a typical HPC cluster, multiple compute nodes work together to complete a computational task.
For example, imagine an application running across four compute nodes:
Node 1
Node 2
Node 3
Node 4
Each node performs part of the calculation. However, they also need to exchange information with one another.
An MPI application might frequently send and receive data between nodes. If communication becomes slow, the entire application can become slower, even when the processors have plenty of available resources.
InfiniBand provides a high speed network for this communication.
A simplified InfiniBand setup includes:
Host Channel Adapter (HCA): The network adapter installed in a compute server. It connects the server to the InfiniBand fabric.
InfiniBand cable or transceiver: The physical connection between the server and the switch.
InfiniBand switch: Connects multiple compute nodes and forwards network traffic between them.
Subnet Manager: Manages the InfiniBand fabric, including discovering devices and configuring routes.
Think of an InfiniBand fabric as a road network connecting different buildings. The compute nodes are the buildings, the cables are the roads, and the switches are the junctions.
If a road develops a problem, traffic might slow down or become disrupted. Similarly, a problem with an InfiniBand link can affect communication between compute nodes.
What happens when an InfiniBand link develops errors?
InfiniBand links continuously exchange data. During transmission, the hardware monitors the integrity of the communication.
If something goes wrong, the hardware can detect errors and update its counters.
These counters help administrators understand what is happening on the network.
For example, a cable or transceiver problem might cause physical transmission errors. The link may attempt to recover automatically. If recovery fails, the link could go down.
The important point is that not every error immediately causes a network outage.
Some errors can occur while the link remains operational. This is why an administrator might see a node online even though its network connection is experiencing problems.
What would an HPC user actually experience?
This is one of the most important parts of troubleshooting.
Users do not normally see InfiniBand counters. They experience the impact on their applications.
Here are some possible symptoms.
1. MPI jobs taking longer than usual
Imagine running an MPI application across eight compute nodes.
Previously, the application completed in 30 minutes. Now it takes 50 minutes, even though the CPU utilisation and memory consumption appear similar.
One possible explanation is that communication between nodes has become slower.
MPI applications frequently exchange data between ranks. If communication is delayed, some ranks may spend more time waiting for messages instead of performing calculations.
This can reduce overall application performance.
However, slow MPI jobs can also result from application behaviour, CPU imbalance, storage delays or other network issues. InfiniBand errors are one possibility to investigate, not an automatic explanation.
2. MPI jobs hanging or waiting for communication
Another possible symptom is an MPI job that appears to stop making progress.
For example, several MPI ranks might be waiting for data from another rank.
If communication between two nodes is disrupted, the application might experience delays or fail to complete a communication operation.
From the user's perspective, the job may appear stuck.
The actual cause could be an application deadlock, an MPI problem or a network issue. Network counters and application logs help narrow down the possibilities.
3. Inconsistent application performance
Sometimes the problem is not permanent.
A job might run normally today but become significantly slower tomorrow. Another job using the same compute nodes might perform differently.
This can happen when a network link experiences intermittent errors or recovery events.
For example, a physical connection might work most of the time but occasionally experience errors that require the link to recover.
The network remains available, but communication performance becomes inconsistent.
4. MPI communication errors or job failures
In more serious situations, network problems can cause communication failures.
Users might see MPI errors, connection failures, communication timeouts or jobs terminating unexpectedly.
The exact error message depends on the MPI implementation, communication library and nature of the failure.
For example, an application using Open MPI and UCX may report communication or transport errors if the underlying connection becomes unavailable.
It is important not to assume that every UCX error is caused by InfiniBand. Configuration mismatches, software problems and other issues can produce similar symptoms.
5. A compute node becoming unavailable
If an InfiniBand link goes down completely, communication through that connection may no longer be possible.
Depending on the cluster configuration and application, this can result in failed jobs, unreachable nodes or communication failures.
The compute node itself might still be accessible through its Ethernet management network. This is another reason why an administrator should not assume that an accessible server has a healthy InfiniBand connection.
Introducing ibqueryerrors
Now that we understand the symptoms, let's look at the tool.
ibqueryerrors is an InfiniBand diagnostic utility that queries and reports port error counters across an InfiniBand fabric.
It is part of the infiniband-diags utilities.
Rather than checking only one network interface, it can examine ports across the fabric and report counters that exceed their configured thresholds.
By default, the threshold is zero. This means counters with nonzero values can appear in the output.
This makes it useful when investigating a suspected network problem because it can help identify which ports are reporting errors.
It can also report additional port and link information, helping an administrator identify the devices involved.
One important thing to understand:
ibqueryerrors identifies counters that deserve investigation. It does not automatically tell you the root cause.
A reported error might be related to a physical connection, a switch port, a network adapter or another condition. Further investigation is needed.
Installing and checking ibqueryerrors
On a RHEL based system, the utility is generally provided by the infiniband-diags package.
You can check whether it is installed:
rpm -q infiniband-diags
If it is not installed, install it using your configured package repositories:
sudo dnf install infiniband-diags
Then check whether the command is available:
which ibqueryerrors
You should also confirm that the InfiniBand devices are visible:
ibstat
The output provides basic information about the local InfiniBand adapter, including its port state and physical link state.
For example, you might see:
CA 'mlx5_0'
CA type: MT4123
Number of ports: 1
Port 1:
State: Active
Physical state: LinkUp
Rate: 200
This is an illustrative example. Actual output depends on the hardware and software version.
An active link is a good starting point, but it does not guarantee that the connection is completely free of errors.
Running ibqueryerrors
The basic command is straightforward:
ibqueryerrors
The tool queries the fabric and reports port error counters that exceed their thresholds.
You might see output similar to the following:
# Port info: Lid 12 port 1
SymbolErrorCounter: 15
LinkErrorRecoveryCounter: 3
LinkDownedCounter: 1
# Port info: Lid 18 port 1
PortRcvErrors: 7
The values above are fictional examples to help explain the output. Counter names and formatting can differ between versions.
What does this information tell us?
It tells us that certain ports have recorded errors or recovery events.
For example, the first port has recorded symbol errors, link recovery events and a link down event.
This gives us a starting point for investigating the affected connection.
To include additional information about the ports and their remote connections, you can use:
ibqueryerrors -r
The -r option reports port information, including details such as the local port, remote port and link information where available.
This can be particularly useful when trying to understand which devices are connected to a problematic port.
Understanding the important InfiniBand error counters
You do not need to memorise every available counter. However, understanding some of the common ones makes troubleshooting much easier.
Let's look at the counters you are most likely to encounter.
1. SymbolErrorCounter
Symbol errors are physical link errors detected during communication.
Think of a cable carrying information between two devices. If the signal is not received correctly, the receiving hardware may detect an error.
Possible causes include:
Poor cable quality, damaged connectors, faulty transceivers or signal integrity problems.
If this counter continues increasing, it deserves investigation.
For example, if a compute node is experiencing slow MPI communication and its connected switch port is also reporting increasing symbol errors, the physical connection becomes a reasonable area to investigate.
2. LinkErrorRecoveryCounter
This counter records how many times the port successfully completed a link error recovery process.
In simple terms, the link experienced a problem and the hardware successfully recovered.
A small historical value does not necessarily indicate a current problem.
However, if the counter continues increasing, it might indicate repeated link instability.
Imagine a network connection that repeatedly loses signal quality and has to recover. Communication may continue, but repeated recovery events can indicate that something is wrong.
3. LinkDownedCounter
This counter records how many times the port failed to recover the link and the link was brought down.
This is more concerning than a simple recovery event.
If the link repeatedly goes down, the connected device might lose InfiniBand connectivity.
Possible causes include physical connection problems, faulty transceivers, switch port issues or hardware problems.
A rising link down counter should be investigated alongside the current port state and switch logs.
4. PortRcvErrors
This counter records packets received with errors.
These can include physical errors, malformed packets and packets discarded because of receive buffer problems.
It is a broader counter, so it does not identify one specific root cause.
If it increases rapidly, investigate the more detailed counters and the affected port.
5. LocalIntegrityErrors
This counter tracks situations where local physical errors exceed a configured threshold.
It can indicate problems with the physical integrity of the link.
If it increases, investigate the connected cable, transceiver, HCA port and switch port.
6. ExcessiveBufferOverrunErrors
This counter is related to receive buffer overruns.
In simple terms, the receiving side cannot handle incoming data within the required flow control conditions.
This can indicate a problem with the link's flow control behaviour or buffer handling.
It should not automatically be interpreted as a faulty cable. The surrounding counters and fabric conditions need to be examined.
7. VL15Dropped
VL15 is a special virtual lane used for InfiniBand subnet management traffic.
The VL15 dropped counter records management packets dropped because of resource limitations.
It is not a direct measure of ordinary MPI application traffic.
If this counter increases, it may indicate a management traffic or resource issue that needs separate investigation.
The meaning and significance of counters can vary with hardware and implementation. Always compare the output with the documentation for the installed adapter and switch.
A practical troubleshooting example
Let's put everything together.
Imagine users report that MPI jobs running between two compute nodes are taking longer than usual.
The cluster administrator checks CPU utilisation, memory utilisation and storage performance. Nothing obvious appears to explain the slowdown.
The administrator suspects a possible InfiniBand issue.
Step 1: Check the local InfiniBand link
On the affected compute node:
ibstat
Confirm that the expected port is active and the physical link is up.
If the port is down, the investigation should start with the link state, physical connection and switch port.
Step 2: Check fabric error counters
Run:
ibqueryerrors
Suppose the output shows increasing symbol errors and link recovery events on a port connected to one of the affected nodes.
This gives the administrator a useful lead.
Step 3: Identify the affected port
Run:
ibqueryerrors -r
Use the reported LID, port and remote port information to identify the connection.
Remember that the LID is a fabric address, not necessarily the Linux device name or server hostname. You may need the fabric's node name mapping or switch management tools to identify the physical device.
Step 4: Check the local hardware counters
On the affected compute node, inspect the available counters:
ls /sys/class/infiniband/mlx5_0/ports/1/counters/
You can then check individual values:
cat /sys/class/infiniband/mlx5_0/ports/1/counters/symbol_error
cat /sys/class/infiniband/mlx5_0/ports/1/counters/link_error_recovery
cat /sys/class/infiniband/mlx5_0/ports/1/counters/link_downed
The device name and port number will depend on the server hardware.
These counters provide another view of the local adapter's port statistics.
Step 5: Compare counter values over time
This is particularly important.
A counter showing 100 errors does not necessarily mean that 100 errors occurred recently.
It might have accumulated over several months.
Record the current values, wait for a suitable interval while the system is experiencing the problem, and check them again.
For example:
Initial symbol errors: 100
After 10 minutes: 100
There has been no increase during that interval.
Now consider:
Initial symbol errors: 100
After 10 minutes: 450
The counter has increased by 350.
That is much more interesting and deserves further investigation.
You should also compare the timing with the affected jobs and application performance.
Step 6: Investigate the physical connection
Once a problematic port has been identified, the administrator can investigate:
- The InfiniBand cable and its connectors.
- The transceiver on both ends of the connection.
- The compute node's HCA port.
- The connected switch port.
- Switch logs and port counters.
- Firmware and hardware health.
If a physical component is suspected, follow the hardware vendor's troubleshooting procedure and maintenance process.
Do not replace hardware simply because a historical counter is nonzero.
Other useful InfiniBand troubleshooting commands
ibqueryerrors is not the only tool available. A few other utilities can help build a clearer picture.
ibstat
ibstat
Displays basic information about the local InfiniBand adapter, including port state, physical state and link rate.
iblinkinfo
iblinkinfo
Displays link information across the InfiniBand fabric. It is useful when investigating connectivity and identifying links between devices.
ibnetdiscover
ibnetdiscover
Discovers devices and connections in the fabric, helping administrators understand the physical and logical topology.
perfquery
perfquery
Queries performance counters for an InfiniBand port. It is useful when investigating individual port statistics.
These tools complement one another. For example, ibqueryerrors can highlight a port with errors, while iblinkinfo and ibnetdiscover can help identify its location within the fabric.
Important: Not every error counter indicates a fault
This is worth emphasising.
A common mistake during InfiniBand troubleshooting is seeing a nonzero counter and immediately assuming that the hardware is faulty.
Many counters are cumulative. They record events over time and may remain nonzero even when the connection is currently healthy.
For example, a link might have experienced a recovery event several weeks ago. The counter will still show that event today.
What matters is understanding the context:
- Is the counter increasing?
- How quickly is it increasing?
- Is the link currently active?
- Are users reporting performance problems?
- Are multiple related counters increasing?
- Does the connected switch port report similar issues?
- Did the problem begin after a hardware or configuration change?
Looking at the overall picture is much more useful than focusing on a single number.
Final thoughts
InfiniBand is a critical part of many HPC environments. When communication between compute nodes becomes unreliable, the impact can extend beyond the network itself.
Users might experience slow MPI jobs, inconsistent application performance, communication delays, hanging jobs or unexpected failures.
For administrators, ibqueryerrors provides a useful starting point for investigating these situations. It helps identify ports reporting errors and gives you a direction for further troubleshooting.
The key is to understand what the counters mean, identify the affected connection and compare counter values over time.
A healthy looking compute node does not always mean a healthy network connection. Sometimes the problem is hidden in the fabric, and knowing how to investigate its counters can save considerable troubleshooting time.
For anyone working with HPC clusters, understanding tools such as ibqueryerrors, ibstat, iblinkinfo and perfquery is a valuable step towards becoming more confident with InfiniBand troubleshooting.
Useful references
Top comments (0)