EP Scaling in MoE Inference: Why Prefill and Decode Diverge

Expert parallelism presents an apparent contradiction: prefill and decode run the same experts, yet they favor very different group sizes. DeepInfer typically uses EP2–EP8 for prefill and EP64–EP128 for decode. The useful starting point is not how many GPUs share the model, but how many rows each expert processes in one step.

The analysis uses DeepInfer’s deployment document and the DeepSeek V4.1-Flash model configuration, and covers only the MoE layer: 384 routed experts, 6 selected per token, plus 1 shared expert; hidden size 5120, expert intermediate size 2304, expert weights in FP4. Hardware numbers are B200 specs (4.5 PFLOPS dense FP8, 8 TB/s memory bandwidth, 900 GB/s NVLink per direction). All times are peak-based estimates per layer per GPU.

At a glance

• Conserved as EP scales: compute per GPU. With the number of tokens per GPU, \(T\), held fixed, each GPU computes \(7T\) rows on average (6 routed experts plus 1 shared expert), independent of EP.
• Changing as EP scales: three quantities. First, the rows each expert gets per step: each GPU holds fewer experts, so the same \(6T\) rows land on fewer experts, one weight read serves more rows, and weight-read time falls. Second, the copies sent per token: the 6 experts spread over more GPUs, so each token is sent more times, approaching 6, and communication time grows. Third, expert load imbalance: with fewer experts per GPU, the extra load of a busy expert is harder to absorb by the other experts on the same GPU, so the busiest GPU falls further behind the average and every other GPU waits for it.
• The two phases differ only in weight-read time. Copies per token and expert load imbalance depend only on EP, not on the tokens per GPU \(T\), so at the same EP both phases spend the same share of compute time on communication and see the same slowdown from the busiest GPU. Rows per expert, however, are proportional to \(T\): prefill has about 32K tokens per GPU, so at EP4 each expert already gets about 2,000 rows and weight reads take about a sixteenth of compute time; decode has only 256 tokens per GPU, so at EP8 each expert gets 32 rows and weight reads take 4 times as long as compute.
• Each phase’s needs set its EP scale. On B200, once an expert gets more than about 150 rows, weight reads stop being the bottleneck and a larger EP saves no more time. Prefill already has about 2,000 rows per expert at EP4, so a larger EP only adds communication and expert load imbalance; it takes the smallest EP that fits in memory. Decode has only 32 rows per expert at EP8 and spends most of its time reading weights; even EP32 gives only 128 rows, and it takes EP64 to cross 150. So decode uses EP64 to EP128, with the NVLink domain carrying the extra communication and EPLB easing expert load imbalance.

1. The deployment puzzle

The deployment recommendations describe two distinct operating points. Prefill takes the smallest expert-parallel group that fits the weights in HBM; decode uses a much wider group. Explaining this difference requires separating useful compute from weight traffic and communication.

It is tempting to explain wide EP as spreading a fixed amount of work over more GPUs. That intuition changes the workload being compared. With requests per GPU held fixed, a larger EP group also serves more requests. The average useful compute per GPU stays constant; the distribution of that work across experts changes.

2. Fixed tokens per GPU imply fixed useful compute

DeepInfer runs attention data-parallel with DP equal to EP, so each GPU serves its own share of requests and holds \(T\) tokens of shape [T, 5120]. The gate selects 6 of the 384 routed experts for each token. Tokens are sent to the GPUs that host their experts; each GPU takes the rows for its own experts, runs two matrix multiplications per expert, and sends the results back, where they are added to the shared expert’s output.

\(T\) is set by the requests on each GPU, so adding GPUs does not make it smaller; the official performance curves are likewise compared at a fixed number of requests per GPU. Under that condition, the whole EP group produces EP × T × 6 rows per layer, and with uniform routing each GPU receives on average

rows, plus \(T\) rows for the shared expert, \(7T\) in total. EP cancels out: going from EP8 to EP64 leaves the useful compute on each GPU unchanged. One row through one expert costs about 70.8 MFLOP, which is about 28 µs per layer for decode at 256 tokens per GPU and about 3.6 ms for prefill at 32,768.

What is conserved is the average useful compute per GPU. With uneven routing the busiest GPU computes more, which is the expert load imbalance discussed in the next section; and when an expert has very few rows, the grouped matrix multiplication pads them to its tile size, so the work actually executed is a little more than the useful compute. That effect is larger for decode at small EP, but it does not change the conclusions here.

Since EP does not reduce compute, its effect has to be found in what it does change.

3. The three quantities EP changes

Rows per expert: what sets weight-read time

What EP changes is how the \(6T\) rows are spread over experts. Each GPU holds 384 / EP experts, and each expert gets

rows per step. For decode with 256 tokens per GPU, EP8 gives each GPU 48 experts with 32 rows each; EP64 gives each GPU 6 experts with 256 rows each.

\(r\) matters because every expert’s weights are read from memory once per step, and \(r\) is how many rows that one read serves. Let an expert have \(P\) parameters. Computing \(r\) rows takes \(2Pr\) operations and reading the weights moves \(Pb\) bytes, where \(b\) is bytes per parameter. Setting the two times equal, \(P\) cancels and gives the weight-reuse threshold:

V4.1’s expert weights are FP4, about 0.53 bytes per parameter with scaling factors. With B200’s specs, \(r^{*} \approx 0.265 \times 562.5 \approx 150\) rows. This counts weights only; including the activations each row reads and writes raises the threshold to about 180 rows, which does not change the conclusions below. Above \(r^{*}\), the expert matrix multiplications are compute-bound; below it, time goes to reading weights. \(r / r^{*}\) is roughly the ratio of compute time to weight-read time, and what a larger EP buys is pushing \(r\) past \(r^{*}\) so that weight reads drop below compute.

The two lines in Figure 1 have the same slope, both growing linearly with EP, but sit 128 times apart. Prefill is far above the threshold already at EP2; decode crosses it only at EP ≈ 38.

Copies per token: what sets communication time

Of a token’s 6 experts, on average \(6 \times (EP - 1) / EP\) are on other GPUs. How many copies a token is sent depends on whether the implementation deduplicates per GPU, and both approaches are in use. Without deduplication, every token–expert pair is sent separately, as in DeepEP’s V1 low-latency mode and in Mega MoE:

With per-GPU deduplication, a token is sent once to each GPU and expanded on arrival, as in DeepEP’s V1 normal mode and V2 direct mode:

Each copy is about 5.4 KB in dispatch (FP8) and about 10.2 KB in combine (BF16). Communication time is proportional to \(T \times c\) and compute time to \(7T\); neither formula for \(c\) involves \(T\), so with or without deduplication the ratio of communication to compute does not depend on \(T\), only on EP and link bandwidth.

In both cases \(c\) grows with EP and approaches 6, but the saving from deduplication fades quickly as EP grows: at EP4 deduplication sends about 45% less (2.47 versus 4.50 copies), at EP64 only 4% less. Deduplication is therefore worth a lot at prefill’s small EP and almost nothing at decode’s large EP. Put the other way, Mega MoE, which does not deduplicate, loses almost nothing in decode but pays extra communication in prefill. DeepInfer uses Mega MoE inside the NVLink domain, so the numbers below assume no deduplication.

Expert load imbalance: how far the busiest GPU falls behind

All GPUs meet at combine, so a layer takes as long as its busiest GPU. If one expert carries \(h\) times the average load, the load on its GPU relative to the average is

The more experts a GPU holds, the more easily one busy expert is absorbed. \(lb\) also depends only on EP, not on \(T\).

Table 1: The three quantities EP changes. Communication / compute assumes Mega MoE without deduplication over NVLink. Prefill uses T = 32,768 per GPU, the DeepInfer benchmark default; decode uses T = 256 per GPU, the official curve’s point at 64 requests per GPU with 4 tokens verified per request per step.

The two EP8 rows in Table 1 make the point: both phases have the same communication-to-compute ratio and the same \(lb\), yet their \(r / r^{*}\) differ by two orders of magnitude. Communication and expert load imbalance depend only on EP; rows per expert depend on EP × T. Figure 3 shows this directly: communication over compute is a single line shared by both phases, while weight reads over compute are two parallel lines 128 times apart.

4. Choosing EP for prefill and decode

Putting weight reads, communication and compute together shows which one is slowest for each combination. Figure 4 marks, for each \(T\) per GPU and EP, which of the three takes longest. On the left, inside the NVLink domain, the official prefill configuration sits in the compute-bound region and the official decode configuration has just left the weight-bound region. On the right, with a 100 GB/s NIC per GPU instead, almost every combination becomes communication-bound, which is why both phases stay inside the NVLink domain.

Prefill: weight reads are no longer the bottleneck, so take the smallest EP

Prefill handles tens of thousands of tokens per GPU per step; at EP4, \(r\) is already more than ten times the threshold and weight reads take about a sixteenth of compute time. A larger EP saves little weight-read time, but communication over compute rises from 0.71 to 0.83 (from 0.39 to 0.61 even with per-GPU deduplication), and expert load imbalance grows too; the officially measured prefill moe_lb is 1.045 at EP2 and 1.074 at EP4. Prefill therefore takes the smallest EP that fits in memory, with the lower bound set by whether each GPU can hold the weights of 384 / EP experts.

Decode: weight reads are the bottleneck, so grow EP until they fall below compute

Decode’s \(T\) is set by concurrency and latency targets, only 256 tokens per GPU. At EP8, \(r\) is just 32 rows: weight reads take about 117 µs and compute only 28 µs. Since \(T\) cannot grow, EP is the only way to push \(r\) past the threshold: going from EP8 to EP64 brings weight reads down to about 19 µs, below compute. This is also how the deployment document explains Wide-EP: it substantially reduces per-step memory traffic in the MoE expert part. The official curves agree (Figure 5): at 64 requests per GPU, EP64 delivers about 15.7K tok/s per GPU and 246 tok/s per user, versus about 12.6K and 198 for EP8.

Once \(r\) passes \(r^{*}\), weight reads are below compute and a larger EP saves no more time: compute stays at 28 µs while communication and expert load imbalance keep growing. By the estimates here, the crossover lies between EP32 and EP64, and the officially recommended EP64 to EP128 sits just past it. In that range communication over compute is about 0.93, every layer synchronizes with 63 or more other GPUs, and one expert at 10 times the average load is enough to push \(lb\) to 2.5. Decode therefore needs an NVLink domain of 64 to 128 GPUs and relies on EPLB to replicate busy experts and rebalance periodically; the officially measured decode moe_lb is 1.090.

Figure 6 lets you check both trade-offs yourself: with \(T\) at 32K, compute is the longest of the three for every EP; with \(T\) at 256, weight-read time drops quickly as EP grows from 8 until \(r\) crosses the threshold, after which communication is close to compute and a larger EP saves no more time.

5. Moving the crossover with hardware and precision

The threshold \(r^{*}\) depends only on weight precision and the hardware’s ratio of compute to memory bandwidth. Figure 7 lists a few common combinations. FP4 weights take half the bytes of FP8 and halve the threshold, which is why V4.1 needs only about 150 rows on B200; with FP4 activations as well, compute doubles and the threshold returns to about 300 rows. H20 has little compute relative to its memory bandwidth, and its threshold is only about 37 rows.

Setting \(r = r^{*}\) gives the smallest EP at which decode crosses the threshold:

At 256 tokens per GPU, FP4 experts on B200 need about EP38, FP8 experts on H100 about EP74, and H20 about EP9. With the same model and load, EP8 to EP16 already gets H20 past the threshold, and a larger EP saves little; H100 needs a larger EP than B200, but an H100 machine has only 8 GPUs, so EP74 must cross nodes and communication becomes the new bottleneck, much like the right side of Figure 4. For another model, replace 384 and 6 with its number of routed experts and experts selected per token.

6. From the cost model to a deployed system

The model uses peak compute and bandwidth, uniform routing, fully overlapped communication, and the MoE layer in isolation. These assumptions make the trade-offs visible. Real utilization, synchronization, and attention costs determine the final latency and throughput.

Matrix multiplications and memory traffic both fall short of peak. The threshold \(r^{*}\) depends on their ratio; at 70% of peak compute and 85% of peak bandwidth it is about 120 rows, and counting each row’s activation traffic raises it by about a fifth. With the threshold anywhere from 120 to 180 rows, decode needs EP between 30 and 45 to cross it: EP8 is far from enough and EP64 is enough, so the conclusion holds.

Routing is not perfectly uniform either. Expert load imbalance makes the busiest GPU compute more and receive more, scaling compute and communication by the same factor and leaving their ratio unchanged; with EPLB enabled, the officially measured factor is between 1.045 and 1.090, a limited effect.

Communication is not always fully hidden either. Every layer has a fixed cost for synchronization and launch that does not depend on how much data moves. For prefill, with about 3.6 ms of compute per layer, it is negligible; for decode, with only about 28 µs, it is a sizable share, and it grows with the number of GPUs that synchronize. This is one more reason decode stops at EP64 to EP128 rather than going further.

Finally, this post covers only the MoE layer. On the official curve, a decode step at 64 requests per GPU takes about 13 ms, of which the MoE layer is only a part; the rest is attention and a large fixed cost that does not depend on batch size. The numbers here are meant for judging which term is the bottleneck at each EP and which way to move EP, not as actual step times; the measured gap between EP64 and EP8 in the official curves points in the same direction as the derivation. How attention and these fixed costs affect the choice of EP is left for later.

The practical rule

The two EP recommendations follow one principle: increase the rows served by each weight read until weight traffic is no longer dominant, then account for the communication and imbalance that wider groups add. Prefill starts beyond that threshold; decode often needs a wider group to reach it.

• Prefill: \(T\) is large enough that \(r\) is already past the threshold, so a larger EP only adds communication and expert load imbalance; take the smallest EP that fits in memory.
• Decode: \(T\) is held down by concurrency and latency, so only EP can push \(r\) past the threshold, until weight-read time falls below compute; use EP64 to EP128, with the NVLink domain carrying the extra communication and EPLB easing expert load imbalance.

The reasoning is not specific to V4.1. DeepSeek’s V3/R1 inference system runs decode at EP144, with the same stated goal of giving each expert a large enough batch; NVIDIA’s Wide-EP on GB200 NVL72 and MegaScale-Infer’s separation of attention and experts onto different GPUs to pool tokens start from the same point. For another model or GPU, the two formulas in Section 5 give a rough sense of which way to move EP.

References

1. DeepInfer, Inference deployment Q&A
2. DeepInfer, README: Performance benchmarks
3. DeepSeek, Day 6: DeepSeek-V3/R1 Inference System Overview
4. NVIDIA, Scaling Large MoE Models with Wide Expert Parallelism on NVL72 Rack Scale Systems
5. Zhu et al., MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism