In a conventional network transfer, the host CPU is the freight handler: data is copied between application memory and kernel buffers, the protocol stack processes every packet, and interrupts punctuate the whole exchange. At 1G this tax was invisible. At 25G and 100G it becomes the bottleneck — a server can saturate its CPUs moving packets for storage traffic while the application starves. RDMA — Remote Direct Memory Access — exists to remove that tax. It is a headline feature of the 100G tier, including adapters in our 100G range, and this guide explains when it earns its complexity.
What RDMA actually does
RDMA lets one machine read and write another machine’s memory directly, with the NICs performing the transfer and the host CPUs involved only in setting it up. The data path bypasses the kernel and avoids the copy chain: application buffer to application buffer, at wire speed, with latency measured in microseconds rather than the tens or hundreds of a kernel stack. Two transports carry it over Ethernet fabrics: RoCE, which runs RDMA over converged Ethernet, and iWARP, which runs it over TCP. RoCE is the common choice in purpose-built data-centre fabrics.
Where it pays
- Disaggregated storage: NVMe-over-Fabrics and clustered file systems, where storage traffic must not tax compute CPUs.
- HPC and AI training: collective operations whose all-reduce traffic is latency-bound and relentless.
- Virtualisation hosts: live migration and storage vMotion at fabric speed.
- Financial and real-time systems: anywhere tail latency, not average latency, is the service level.
What it demands in return
RDMA is not a checkbox; it is a fabric discipline. RoCE wants a lossless-ish Ethernet: priority flow control, congestion management and carefully matched settings on every switch and NIC in the path. Misconfigured, it performs worse than the conventional stack it replaced. The NIC must implement the transport in silicon — software RDMA is a contradiction — and the operating system and application must be RDMA-aware: your storage stack or MPI library is the consumer, not a generic socket application. Budget for the design work; it is real, and it is where projects succeed or quietly revert to TCP.
Low latency beyond RDMA
Even without RDMA, latency engineering separates serious adapters from commodity ones: cut-through behaviour, interrupt moderation done right, offloads that keep the fast path fast. Our standard 100G adapter design emphasises low latency and RDMA together, because the buyers of 100G cards are precisely the buyers whose workloads notice microseconds. If your workload is throughput-bound bulk transfer — backup, replication windows measured in hours — plain adapters serve perfectly and RDMA’s discipline buys nothing. Latency, like bandwidth, should be bought where the application can prove it feels it.
A pragmatic adoption path
Start with one bounded workload — a storage cluster is the classic first citizen — on its own fabric segment, tuned end-to-end. Measure CPU per gigabyte moved before and after; that number, not the latency headline, usually makes the business case. Expand by workload, keeping the conventional fabric for everything that does not benefit. And describe the workload when you inquire: "RDMA" on a request gets you a controller conversation, not a catalogue page, which is as it should be.
Does RDMA require special switches?
For RoCE, the fabric needs lossless Ethernet features (PFC, ECN) configured consistently. It runs on data-centre-class switches, but they must be designed-in, not defaulted-in.
Can RDMA and normal traffic share a fabric?
Yes, with proper traffic-class separation — that is what "converged" means. The design must protect conventional traffic from the lossless classes and vice versa.
Measuring whether RDMA helped
RDMA projects should be judged on evidence, and the evidence is specific. CPU utilisation per gigabyte moved is the headline metric: a successful deployment collapses it. Tail latency of the storage or collective operation — the 99th percentile, not the average — is the second. The third is deceptively simple: did the application’s own throughput rise, or did the bottleneck merely move to the disks, the GPUs or the code? Measure all three before and after, on the same workload, or the project will be argued about rather than concluded. Our team can help frame the test when a deployment starts from our 100G adapters — the numbers, not the narrative, close the case.
One more discipline: keep the conventional path healthy. RDMA fabrics carry the workloads that provably benefit; everything else stays on the standard stack, which remains simpler to operate and debug. The goal is a data centre with an express lane, not a data centre that bet everything on one transport.
Is RDMA only for huge clusters?
No — a two-node storage pair can benefit measurably. The threshold is not cluster size; it is whether CPU-per-gigabyte or tail latency is a proven constraint in your workload.
RoCE versus iWARP, without the theology
The two Ethernet RDMA transports differ in where they put their trust. RoCE rides on UDP and asks the fabric to behave — lossless classes, priority flow control, congestion signalling — rewarding a purpose-built fabric with the lowest latency and the least CPU. iWARP rides on TCP and asks the NIC to be cleverer, tolerating an ordinary routed fabric at some cost in latency and processor involvement. The industry has largely standardised on RoCE inside designed data-centre fabrics, which is where RDMA is deployed deliberately; iWARP survives where RDMA must cross fabrics nobody controls. For a buyer, the transport question is really a fabric question: if you own and can tune the whole path, RoCE is the mainstream answer; if the path crosses networks you do not govern, have a longer conversation with whoever does. Either way, the NIC must implement the chosen transport in silicon — confirm it by name in the specification, not by the word "RDMA" in a brochure headline.
If this guide leaves one impression, let it be that RDMA is a systems purchase wearing a network card’s clothes. The card is where the budget line sits; the fabric, the switches, the tuning and the measurement culture are where the result is decided. Buy the system, and the card will justify itself in the first benchmark.
One more population deserves a mention: the teams who should not buy RDMA yet. If the fabric is shared with uncontrolled traffic, if nobody on staff has owned PFC tuning before, or if the workload has not been profiled to prove the network is its constraint, the correct purchase is a profiling exercise and a fabric design — after which the NIC decision becomes easy. RDMA rewards the prepared and bills the hopeful.
Prepared teams also document the tuning they applied — queue pairs, MTU, congestion settings — so the fabric’s behaviour survives staff changes.