AMD adds GPU-initiated SDMA copy engines to RCCL and rocSHMEM

Overview

Updated 46 minutes ago

AMD now lets a running GPU kernel hand large data transfers to dedicated System Direct Memory Access (SDMA) copy engines, instead of having compute units (CUs) execute every load and store. The feature is in RCCL, AMD's ROCm Communication Collectives Library, and rocSHMEM, its GPU networking library. Both are part of the ROCm software stack, and the feature is available in version 10.1.

A kernel writes DMA commands into a ring buffer and rings a doorbell; the SDMA engine moves the data. On an eight-GPU AMD Instinct MI350X node, AMD measured large-message AllToAll and AllGather bandwidth. It was comparable to or slightly above the default RCCL path, using one CU instead of 64.

Why it matters

Freeing 63 of 64 compute units during large transfers can give inference workloads more GPU capacity for the math that generates tokens.

Questions about this story

Free account needed to ask — your question is kept and asked for you right after sign-up. Answers are public.

No questions yet — be the first to ask.

Key Indicators

Compute units used for large-message GPU-initiated SDMA transfers

AMD's MI350X tests used one CU to issue SDMA commands.

Compute units used by the default RCCL AllToAll and AllGather path

The load/store path assigns the copying work to 64 CUs in AMD's tests.

32 MiB

AllToAll message size where RCCL switches to GIN-SDMA

Smaller messages use vector load-store operations.

512 KiB

AllGather message size where RCCL switches to GIN-SDMA

AMD reports one CU is enough above this threshold.

128 KiB

Smallest message size where rocSHMEM uses SDMA

Below this, rocSHMEM stays on the load-store path.

GPUs in AMD's MI350X test node

All benchmarks were run within one node.

Voices

Curated perspectives — historical figures and your fellow readers.

People Involved

Aurelien Bouteiller

Co-author, AMD GPU-initiated SDMA announcement

Co-authored the November 2026 announcement

Organizations Involved

Advanced Micro Devices

Semiconductor Company

Develops ROCm, RCCL and rocSHMEM; announced the feature

The chipmaker that develops the Instinct GPUs and the ROCm software stack, including RCCL and rocSHMEM.

ROCm Communication Collectives Library (RCCL)

GPU communication library

Added a GPU-initiated SDMA backend, enabled with NCCL_GIN_TYPE=7

ROCm's library for collective operations such as AllToAll and AllGather across GPUs.

rocSHMEM

GPU networking library

Supports GPU-initiated SDMA for put operations and all-to-all

An intra-kernel networking library with an OpenSHMEM-like interface.

Timeline

1 event Latest: Today

November 2026
AMD details GPU-initiated SDMA for RCCL and rocSHMEM
Today Release

Ships in ROCm 10.1; matches default RCCL large-message bandwidth with one CU instead of 64.
Today

Scenarios

rocSHMEM extends SDMA to messages under 128 KiB

Possible Resolves by Q2 2027

Discussed by: AMD's announcement, which identifies 128 KiB as the current cutoff

The blog says rocSHMEM does not use SDMA for messages smaller than 128 KiB. A later release could lower that cutoff, putting small put operations on the copy-engine path and changing how small-message kernels are tuned.

RCCL makes GIN-SDMA the default above 32 MiB

Possible Resolves by End of 2027

Discussed by: AMD's benchmark description; the blog gives no default-on commitment

Today the SDMA backend is opt-in through the environment variable NCCL_GIN_TYPE=7. If AMD makes it the default for large AllToAll, every RCCL user on supported hardware gets the one-CU path without changing code.

An inference framework adopts the SDMA path for tensor-parallel collectives

Uncertain Resolves by Q3 2027

Discussed by: AMD's positioning of the feature for inference; no framework has announced support

Inference engines use AllToAll and AllGather during tensor and expert parallelism. Adoption would appear as a release note or merged pull request using RCCL's GIN-SDMA backend in a serving stack.

Historical Context

3 moments from history that rhyme with this story — and how they unfolded.

1981

IBM PC DMA controller (1981)

IBM's first personal computer used the Intel 8237 DMA controller, a chip that could move data between memory and peripherals without the CPU reading and writing each byte. Programmers set up the transfer. The controller did the copying.

Then

The CPU spent less time on I/O, and floppy and other peripheral transfers became practical on desktop machines.

Now

DMA controllers became standard in PCs and embedded systems. The pattern of a dedicated engine handling bulk data persists.

Why this matters now

AMD's SDMA engines follow the same division of labor. A general-purpose processor issues commands, and a specialized engine moves the bytes.

2016

NVIDIA GPUDirect Async (2016)

NVIDIA and Mellanox demonstrated GPUDirect Async, which let a CUDA kernel trigger InfiniBand transfers directly instead of asking the host CPU to post work to the network card. The GPU could start communication while it continued computing.

Then

High-performance computing applications used it to cut latency and overlap communication with computation.

Now

GPU-initiated networking became a feature of high-performance interconnects and later GPU communication libraries.

Why this matters now

It established the idea that a running kernel, not the CPU, should control data movement. AMD's release applies that idea to intra-node copy engines.

2022

NVIDIA Hopper Tensor Memory Accelerator (2022)

NVIDIA's Hopper architecture added the Tensor Memory Accelerator, a dedicated unit that copies tiles between global memory and shared memory asynchronously. Threads issue a descriptor and continue working while the copy runs.

Then

Matrix-multiply and attention kernels used TMA to reduce register pressure and free threads for math.

Now

TMA became a core part of Hopper kernel design and influenced later GPU programming models.

Why this matters now

It shows that moving copies onto dedicated hardware changes kernel design, not just raw bandwidth. AMD is making the same bet for inter-GPU transfers.