Multi-GPU inference hangs or dies with NCCL timeouts: shared memory, P2P, topology and network interface problems

#Symptoms

• The server starts loading a model across several GPUs (--tensor-parallel-size 4) and hangs with no error, GPUs at 100% utilisation or idle.
• After minutes: Watchdog caught collective operation timeout: WorkNCCL(SeqNum=..., OpType=ALLREDUCE ...) ran for 600000 milliseconds before timing out.
• NCCL error: unhandled system error, NCCL WARN ... failed to open shared memory or Bus error in container logs.
• Works with 1 GPU, fails with 2+; or works on one host type and not another.

Tensor parallelism splits each layer across GPUs, so every forward pass needs collective operations (all-reduce) over NCCL. If GPUs can't exchange data, everything waits.

#Step 1: turn on NCCL logging

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,ENV,NET,P2P,SHM

Restart and read the first lines NCCL prints:

• Which transport it chose between GPUs (P2P/CUMEM, P2P/IPC, SHM, NET/Socket, NET/IB).
• Which network interface it uses (NET/Socket : Using [0]eth0:10.0.0.5).
• Any WARN lines: they usually name the problem.

#Step 2: shared memory in containers

NCCL and the serving framework use /dev/shm for intra-host communication. Docker's default is 64 MB, which is far too small.

docker run --gpus all --ipc=host ...          # what vLLM's docs recommend
# or
docker run --gpus all --shm-size=16g ...

On Kubernetes, mount a memory-backed emptyDir at /dev/shm:

volumes:
  - name: dshm
    emptyDir:
      medium: Memory
      sizeLimit: 16Gi
containers:
  - name: vllm
    volumeMounts:
      - name: dshm
        mountPath: /dev/shm

Memory-backed emptyDir counts against the pod's memory limit; size limits accordingly.

#Step 3: GPU topology and P2P

nvidia-smi topo -m
nvidia-smi nvlink --status

NV# entries mean NVLink between GPUs (fast, reliable). PIX, PXB, PHB, NODE, SYS mean PCIe paths of increasing distance; SYS crosses CPU sockets.

Known problems:

Quick test whether P2P is the issue:

NCCL_P2P_DISABLE=1 python -m vllm.entrypoints.openai.api_server --model ... --tensor-parallel-size 4

If it now works, fix P2P properly rather than leaving it disabled.

#Step 4: network interface selection (multi-node or odd hosts)

Even on a single host, NCCL may try to bootstrap over an interface that doesn't work (docker bridge, a down interface, IPv6 only). For multi-node (pipeline/tensor parallel across hosts) it matters a lot.

export NCCL_SOCKET_IFNAME=eth0          # or ^docker,lo to exclude
export GLOO_SOCKET_IFNAME=eth0          # PyTorch's Gloo backend used for some coordination
export NCCL_IB_DISABLE=1                # only if no InfiniBand/RoCE is present and NCCL tries it

For InfiniBand/RoCE clusters, check the HCA is visible in the container (ibv_devinfo), and that the right devices and network policies are configured (NCCL_IB_HCA). Cloud GPU clusters have provider-specific NCCL plugins and settings (for example for EFA on AWS, or InfiniBand SKUs on Azure); use the provider's documented environment.

#Step 5: prove it with nccl-tests

Take the model server out of the picture:

git clone https://github.com/NVIDIA/nccl-tests && cd nccl-tests && make
./build/all_reduce_perf -b 8M -e 1G -f 2 -g 4

It should finish quickly and report bus bandwidth close to the expected NVLink/PCIe numbers. A hang here confirms an infrastructure problem, not a model server bug.

#Step 6: hangs under load, not at startup

• One rank crashed: if one worker process dies (OOM, Xid error), the others wait in the collective until timeout. Check all ranks' logs and dmesg/DCGM for Xid errors.
• GPU memory pressure: OOM in one rank during a spike; lower --gpu-memory-utilization or --max-num-seqs.
• Timeouts too short for big loads: very large weight loads can exceed default init timeouts; raise framework init timeouts only after ruling out real hangs.
• PyTorch settings TORCH_NCCL_ASYNC_ERROR_HANDLING=1 (default in recent versions) make hung collectives fail instead of hang forever, so the pod restarts.

#Decision table

#Prevent it next time

• Bake --ipc=host or /dev/shm sizing into every GPU deployment template.
• Run nccl-tests as part of GPU node acceptance and after driver upgrades.
• Keep NCCL_DEBUG=WARN on in production and ship logs.
• Pin TP groups to GPUs with NVLink or the closest PCIe topology.