It’s 2:17 AM.
Your dashboard is green.
CPU usage is low.
Latency graphs look… too good to be true.
Yet your distributed database is pushing hundreds of gigabytes per second across machines.
No magic.
No cheating.
Just RDMA.
first, the uncomfortable truth
Most of us learned networking like this:
“Send data → TCP/IP → kernel → CPU → copy buffers → receive → copy again”
And we accepted it.
But modern systems looked at this and said:
“Why are CPUs doing delivery jobs?”
And that’s where RDMA was born.
What is RDMA ?
RDMA (Remote Direct Memory Access) allows one machine to directly read or write the memory of another machine over the network without involving the remote CPU or OS kernel.
Yes, thats right.
Memory-to-memory communication over the network.
How networking traditionally works ?
When you send data over TCP:
- App → Kernel
- Kernel → Network stack
- CPU handles interrupts
- Data copied multiple times
- Receiver CPU wakes up
- Kernel copies again
- App finally gets data
Result:
- Good latency
- CPU overhead
- Cache pollution
- Context switching hell
How RDMA works ?
With RDMA:
- App tells NIC where memory is
- NIC sends data directly
- Remote NIC writes straight into RAM
- Remote CPU is not disturbed
Result 🚀:
- Zero-copy
- Sub-microsecond latency
- Near-zero CPU usage
- Line-rate throughput (100–400 Gbps)
The “wait… what?” moment
The remote CPU doesn’t even know data arrived.
No syscall.
No interrupt.
No kernel involvement.
This is called one-sided communication.
Key RDMA concepts
1. Memory Registration
- Memory is “pinned”
- NIC gets permission to access it
- Prevents OS from moving it
2. Queue Pairs (QP)
- Send Queue + Receive Queue
- Managed entirely by hardware
3. One-sided ops
- RDMA WRITE → write remote memory
- RDMA READ → read remote memory
- RDMA SEND/RECV → like messaging, but faster
RDMA vs TCP
| Feature | TCP/IP | RDMA |
|---|---|---|
| CPU usage | High | Almost zero |
| Memory copies | Many | Zero |
| Latency | ~10–100 µs | < 1 µs |
| Throughput | Limited | Line rate |
| Kernel involvement | Required | Bypassed |
Where RDMA is ACTUALLY used
This isn’t theory. RDMA powers real systems:
Distributed Databases
- In-memory DBs
- Sharded OLTP engines
Storage
- NVMe over Fabrics
- Distributed file systems
AI / ML
- GPU-to-GPU communication
- Large model training
Finance
- High-frequency trading
- Real-time risk engines
If something is fast + distributed, RDMA is probably underneath.
RDMA doesn’t mean “new cables”
RDMA runs over existing networks:
| Technology | Runs On |
|---|---|
| InfiniBand | Native RDMA |
| RoCE | Ethernet |
| iWARP | TCP/IP |
Most cloud data centers use RoCE today.
RoCE Header format-
Examples of RDMA NICs:
Intel: Ethernet 810/820 Series (iWARP & RoCEv2).
Marvell: FastLinQ (Universal RDMA, RoCE & iWARP).
Chelsio: Known for high-performance RDMA adapters.
Why RDMA isn’t everywhere
Because RDMA is powerful but dangerous:
- Requires special NICs
- Needs lossless networks
- Hard to debug
- Memory bugs = catastrophic
- Not beginner-friendly
RDMA is like giving your NIC root access to RAM.
So, Handle with care
Even if you never write RDMA code:
- Your databases use it
- Your cloud storage depends on it
- Your AI workloads require it
- Your latency SLAs benefit from it
Understanding RDMA explains:
- Why cloud infra feels “unreasonably fast”
- Why CPUs aren’t the bottleneck anymore
- Why networking is becoming hardware-driven
The mental model that sticks
TCP/IP =
“Please OS, copy this data safely and let me know”RDMA =
“Hey NIC, write directly into that memory address over there”
One-line takeaway
RDMA is how modern data centers move data at near-memory speed across machines.
RDMA isn’t hype.
It’s invisible infrastructure. the kind that changes everything quietly.
The next time you see:
- insane throughput
- suspiciously low CPU
- lightning-fast distributed systems
You’ll know.
RDMA is working.
References -
https://wiki.debian.org/RDMA
https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet
https://resource.fs.com/mall/resource/intel-e810-series-ethernet-network-adapters-datasheet-20251125121144.pdf
Top comments (0)