NIMBLE Performance: DeepSeek-V3's MoE Layer in SGLang and vLLM
Machine Specifications: TACC Horizon
| CPU Model | CPU Core Info | Memory | GPU Model | GPU Memory | Interconnects |
|---|---|---|---|---|---|
| NVIDIA Grace (Arm Neoverse V2) | 2x72 cores @3.4 GHz | 960 GB | NVIDIA GB200 (4/Node, NVL4) | 185 Gib | NVIDIA ConnectX-8 InfiniBand XDR (800 Gbps), 2/Node(400 Gb/s) |
NIMBLE plans each AllToAllV from the byte matrix the ranks pass, sending each transfer directly or relaying it through another GPU of its node onto a less loaded NIC. The charts time DeepSeek-V3's MoE layer (FP8, prefill) on the real model at expert-parallel sizes (EP) 8, 16 and 32, on 2, 4 and 8 GB200 NVL4 nodes: the whole layer, its communication (dispatch and combine) and its compute, which is the same for all three.
- NCCL: the same MoE dispatcher over all_to_all_single on NCCL, so only the transport differs.
- NIMBLE: the fused mode, one AllToAllV per direction, moving the same bytes as NCCL.
- NIMBLE dedup: a token crosses the network to a node once and fans out there over NVLink.
Software: SGLang at commit fbf8f1d, vLLM 0.30.0, PyTorch 2.13 (CUDA 13), CUDA 13.2 and NCCL 2.30.7, for all three.
SGLang
DeepSeek-V3 MoE layer in SGLang: NCCL, NIMBLE and NIMBLE dedup
At 16K tokens per rank the MoE layer is 1.21-1.28x faster with NIMBLE than with NCCL, and 1.32-1.53x with dedup (EP 32: 122.7 ms with NCCL, 96.1 ms with NIMBLE, 80.4 ms with dedup). Only dedup's smallest batch at EP 8 is slower (1K tokens: 4.36 against 3.83 ms).
vLLm
DeepSeek-V3 MoE layer in vLLM: NCCL, NIMBLE and NIMBLE dedup
NIMBLE is faster than NCCL at every batch and EP size, 1.09-1.24x on the layer, and dedup reaches 1.20-1.34x at the largest batches. The gains are smaller than in SGLang because vLLM's expert kernel runs untuned at large batches, so compute is a larger share of the layer.
To run these yourself: the serving examples, which run SGLang's server and offline engine, vLLM's server and a client, with step-by-step commands and the serving guide
