DeepEP Dispatch: Four Designs for Unknown Receive Counts

On this page

• At a glance
• 1. Separate completion, placement, and sizing
• 2. Let workload and topology set the constraints
• 3. V1 normal: resolve counts before transfer
• 4. V1 low-latency: reserve capacity and signal completion
• 5. V2 direct: decouple counts from completion
• 6. V2 hybrid: reduce traffic at the node boundary
• 7. Compare the protocols on the same axes
• 8. Turn traffic into a latency estimate
• 9. Account for the SM and buffer trade-offs
• Choosing a dispatch design

MoE dispatch looks like an all-to-all, but its receive sizes are not known when the operation begins. The gate determines token destinations at runtime, and the routing can change at every layer and step. Unlike MPI_Alltoallv, the receiver cannot begin with a complete table of incoming counts.

DeepEP offers four answers: V1 normal, V1 low-latency, V2 direct, and V2 hybrid. Their differences become clearer when each is asked the same questions: who owns a buffer segment, how is completion detected, when are output positions known, and how many copies cross the bottleneck link?

The analysis uses the DeepEP source code, using the last V1 commit 29d31c0 and the current V2 commit f6aa38f, and covers dispatch only. Examples use hidden size 7168, 8 experts per token and 8 GPUs per node. This is the DeepSeek-V3 configuration on H800. The official DeepEP V1 latency table and the ISCA 2025 paper both use these numbers, which makes calibration possible.

At a glance

• One problem, four designs. V1 normal mode exchanges counts first and sends data afterwards. V1 low-latency mode reserves buffers for the worst case, and the count, sent last, also signals that the data has arrived. V2 reserves buffers too, but uses a barrier to detect arrival and a copy epilogue to place the data, so counts and data can travel at the same time. Hybrid adds per-node deduplication and two-hop forwarding on top of this.
• The designs differ because of load and network. Prefill has large batches and cares mostly about bandwidth. Decode has small batches, needs CUDA graph and cannot wait for the CPU. Meanwhile, NVLink inside a node is much faster than IB between nodes.
• When decode crosses nodes, most of the time goes to the NIC. A simple byte model fits the official latency table for V1 low-latency mode, and at EP128 the bandwidth term is about 80% of the total. As a rough estimate, hybrid cuts this term by somewhere between 17% and a half, depending on routing.
• V2 no longer supports zero-SM EP. Judging from the mechanism, hybrid forwarding and NVLink transfers both need SMs, so those SMs can’t go back to computation while data is still on its way.

1. Separate completion, placement, and sizing

A remote write puts bytes into the receiver’s memory; it does not by itself tell the receiving kernel which slots are valid. Here, a record means one token’s data sent to one destination. Safely consuming those records requires both a placement rule and a completion protocol.

The receiver needs three pieces of information: whether all expected records have arrived, where each record belongs in the expert input, and how many rows the output must contain. Completion, placement, and output sizing are separate problems even when one mechanism helps solve several of them.

Three kinds of numbers are involved, and they are easy to confuse:

2. Let workload and topology set the constraints

DeepSeek’s production system deploys prefill and decode separately and gives them different EP sizes (ISCA 2025 paper, §2.3.1). Prefill has large batches, so bandwidth is the bottleneck and fixed overhead gets amortized. An LMSYS post also notes that CUDA graph is typically off in this phase. Decode has small batches, so kernel launches and waiting take up a large share of the time. It therefore uses CUDA graph, and capturing a graph requires fixed shapes and no waiting on the CPU.

On the network side, GPUs in the same NVLink domain can write to each other directly, while traffic between nodes has to go through the NIC. The paper’s H800 numbers put NVLink at about 4 times the bandwidth of IB (§4.3).

The four designs map onto these two conditions as follows:

V1 first splits into two sets of kernels by load. V2 splits only by network into direct and hybrid, and leaves the differences in load to parameters.

3. V1 normal: resolve counts before transfer

V1 normal resolves uncertainty before moving the payload. It first exchanges counts, then allocates the output, and finally transfers the data. This makes exact sizing straightforward, at the cost of an explicit synchronization point.

The first step is counting. A kernel started by get_dispatch_layout scans the top-k and computes the token counts per GPU, per node and per expert, along with a token-to-GPU table. Then notify_dispatch exchanges these counts, writes the totals into pinned memory the CPU can read, and computes prefix-sum matrices in GPU memory.

The second step is allocation. The CPU polls the pinned memory until the counts arrive, then allocates an output of the exact size.

Only the third step sends data. The data kernel is organized into channels. A channel takes two SMs and handles a contiguous range of tokens. Each channel starts by sending its start and end positions to the receiver. The receiver adds the source GPU’s prefix sum to the channel start to get the first row, then increments it for each record and writes the data straight into the output. The output is ordered by source GPU and channel, and grouping by expert is left to the caller.

The data takes two hops. Across nodes, each node gets one copy, sent to the GPU with the same index on the target node, which then forwards it over NVLink to each target GPU. The buffers in between are ring queues. When a queue is full, the sender waits for the head to advance, so buffer size doesn’t depend on the total token count. The V1 README also notes that this design saves memory but adds complexity and potential deadlocks.

The price is CPU synchronization: an extra round of count exchange before any data moves, plus the GPU waiting on the CPU. Output shapes also differ every time, which LMSYS describes as “symbolic shapes that are incompatible with CUDA Graph”. The num_worst_tokens option can skip the CPU sync, but the docs say it is for intranode only.

4. V1 low-latency: reserve capacity and signal completion

Decode cannot wait for the CPU, so it cannot exchange counts in advance either. Low-latency mode instead reserves buffers for the worst case and gives each segment a single writer. The receive buffer has this shape:

[local expert, source GPU, Tmax, one 7408 B message]

The data from source GPU r to expert e has its own segment, sized for Tmax. Only r writes to this segment, so r can assign slots with a local atomic add, with no coordination between GPUs.

One kernel has a send phase and a receive phase.

In the send phase, data warps send each token to the segment of each of its top-k experts, using RDMA across nodes and direct NVLink writes within a node. Meanwhile, the last warp of each SM scans the top-k and counts the records going to each expert. Once all requests for an expert have been issued, it writes the count into the remote count table. The value written is −n−1, so even n = 0 doesn’t write a zero.

Counting can happen alongside the data, but the count must be sent last, because the receiver takes the count to mean the data has arrived. The sender doesn’t know whether the data has landed. It relies on channel ordering:

In the receive phase, the receiver polls the count table until it reads a non-zero value. It then uses an atomic add to get where this batch starts in that expert’s output, and copies and compacts the data. The per-expert counts go to masked GEMM.

This design has one more benefit. With return_recv_hook on, the kernel runs only the send phase, and the receive phase starts when the hook is called. Over RDMA, the NIC moves the data while it is in flight, so no SM is occupied.

Its drawbacks are the lack of deduplication and the memory use. A token going to two experts on the same GPU is sent twice, so every token is sent 8 times. Each expert also needs a worst-case buffer, which is acceptable for decode with its small Tmax but far too large at prefill scale.

5. V2 direct: decouple counts from completion

V2 uses one set of kernels for both kinds of load. It has to work without waiting for the CPU, deduplicate per GPU, and still give prefill an exact output size. Its receive buffer is partitioned by source GPU, dropping the expert dimension:

[source GPU, Tmax, one record]

Only source GPU r writes to its own segment, so slots still come from a local atomic add. Two groups of warps work at the same time inside one kernel.

Notify warps do the counting. They scan the top-k and accumulate per expert and per GPU, deduplicating before the per-GPU count. After the SMs’ counts are combined, SM0 sends them to each target GPU. SM0 then waits for the counts from other GPUs, writes the totals to host memory if CPU sync is on, and finally computes two prefix sums: the cumulative records from each source GPU, and the first output row of each expert.

Dispatch warps send the data. Each record carries its top-k and source index, so a receiver can tell which experts it belongs to just by reading it. Before sending, the warp deduplicates per GPU and gets a slot with a local atomic add. The NVLink path uses TMA to write directly to remote GPU memory. The RDMA path first writes to a registered send buffer, and then the NIC sends it.

When both groups are done, the kernel reaches an end barrier and then triggers the copy epilogue. The copy epilogue uses prefix sums to find each record in the buffer and reads its top-k. For each expert on this GPU, it does an atomic add to get the output row.

As a result, the count no longer has to signal arrival. The barrier detects arrival, the copy epilogue places the data, and the CPU reads the output size if it needs it. Whether the counts arrive before or after the data makes no difference to the result.

6. V2 hybrid: reduce traffic at the node boundary

Direct mode only deduplicates per GPU, which saves little at large EP. Under uniform routing at EP128, a token’s 8 experts land on 7.89 different GPUs on average, almost the same as sending 8 copies.

Section 4.3 of the paper offers another route: tokens going to the same node are sent once over IB and then forwarded over NVLink. The IB cost then depends only on the number of target nodes M, falling from 8t to Mt. DeepSeek-V3 was designed around this with node-limited routing. It splits 256 experts into 8 groups, places each group on one node, and guarantees that a token goes to at most 4 nodes. Hybrid follows this path:

The first hop crosses nodes. One warp corresponds to one channel, and tokens are assigned to channels in turn. Each token is deduplicated per node, written to the send buffer, and sent through the Rail team to the GPU with the same index on the target node. Data for the local node skips the NIC.

The second hop stays inside the node. A forward warp on the relay GPU starts as soon as it sees the tail advance, without waiting for the first hop to finish. It reads the top-k, deduplicates per GPU, and uses TMA to write to the target GPUs over NVLink.

Counts take the same two hops. Per-GPU and per-expert counts go through Rail to the GPU with the same index, which sums them and sends the results over NVLink to the GPUs in its node.

The two hops have different writers, so they assign slots differently:

The tail here serves a different purpose from the progress pointers in V1 normal mode. V1 uses them for flow control. Hybrid’s buffers are reserved for the worst case and never wrap, so its tail exists to let the second hop start early.

7. Compare the protocols on the same axes

Two structural rules emerge from the comparison. A single-hop path can deduplicate per GPU; deduplicating per node requires a relay. Slot allocation is a separate question: local counters suffice when each remote segment has one writer, while shared segments require coordination among their writers.

8. Turn traffic into a latency estimate

To connect protocol design to latency, count the bytes crossing the limiting port and separate their transfer time from fixed overhead:

communication time ≈ fixed overhead + bytes at the bottleneck port ÷ effective bandwidth
bytes at the bottleneck port = tokens × copies × record size

Start by calibrating against the official table for V1 low-latency mode. The test used H800 and CX7, with 128 tokens per GPU and 7408 B per message. Traffic within a node goes over NVLink, so only (EP−8)/EP of the bytes pass through the NIC:

The fit gives a fixed overhead of about 35 µs and an effective bandwidth of about 46 GB/s. At EP128 the bandwidth term is about 155 µs, 80% of the total, so NIC bytes are the main bottleneck in this configuration. The ratio of combine to dispatch latency is 1.92 (369 ÷ 192), and the ratio of their message sizes is 1.94 (14336 ÷ 7408), which points the same way.

Section 2.3.2 of the paper uses the same arithmetic: dispatch plus combine is (1 B + 2 B) × 32 × 9 × 7 KB ÷ 50 GB/s, about 121 µs. The 9 copies are 8 routed experts without deduplication plus 1 shared expert. Section 4.3 takes 40 GB/s as the effective IB bandwidth, close to the fit above.

The same model gives a rough estimate for hybrid. A V2 record is 7552 B. The premise of node-limited routing matters here: the paper assumes expert groups are deployed per node and discusses training, while the global policy that the official EPLB uses for decode ignores expert groups. So for decode at EP128, we can’t simply assume a token goes to at most 4 nodes. Below are estimates under two routing assumptions, with 16 nodes at EP128:

The estimated reduction ranges from about 17% to roughly one half because the two routing assumptions send different numbers of copies through the NIC. This is a bandwidth-term estimate. Actual latency also includes the relay hop, barriers, and routing-dependent overhead.

9. Account for the SM and buffer trade-offs

The V2 README says buffers are larger than in V1 and that 0-SM RDMA low-latency EP is no longer supported, without explaining the latter.

Zero SM requires that only the NIC is working while data is in flight. Judging from the mechanism, two kinds of V2 work can’t meet this. One is NVLink transfer, where SMs perform the writes, so nothing keeps writing once the kernel exits. The V1 docs also say that the low-latency NVLink path is “somehow incompatible” with hooks. The other is hybrid forwarding, which reads the top-k, deduplicates and writes to target GPUs token by token, none of which a NIC can do. Section 4.4.1 of the paper also notes that message handling and NVLink forwarding both occupy SMs, with up to 20 SMs on H800 used for communication during training.

Waiting for counts, the end barrier and the copy epilogue are not fundamental obstacles. They could move to after the hook, as the V1 receive phase does. So a pure RDMA direct path could in principle be split into two phases as well, but V2 doesn’t provide one.

In exchange, the official numbers are up to 1.3x peak performance and up to 4x fewer SMs, with only 6 SMs for the 8×4 topology.

Choosing a dispatch design

Beyond these four designs, when DeepInfer deploys V4.1 it uses Mega MoE if the EP group is inside an NVLink domain, and DeepEP only across nodes. Mega MoE faces the same unknown receive counts: it exchanges counts inside the kernel, then has the receiver pull the data, as covered in the previous post.