EPLB activated experts vs token balance

EPLB activated experts vs token balance is a choice between two load proxies. Equal token counts help when expert compute grows with tokens. Fewer activated replicas help when loading expert weights sets the decode cost.

The objectives can disagree even with the same expert placement. Splitting one expert's tokens across two replicas improves token balance only if the split fixes a bottleneck. It always activates another weight copy.

Measure the grouped-GEMM regime before changing the router. Prefill, decode, hot experts, and cold experts need not sit on the same side of the crossover.

EPLB activated experts vs token balance

Expert parallelism places different routed experts on different GPUs. A popular logical expert can have redundant physical replicas so more than one GPU can process its tokens.

The DeepSeek EPLB repository computes which experts to replicate and where to place each physical copy from estimated expert loads.

Current vLLM EPLB documentation describes the same control loop. It records loads, periodically rearranges expert mappings, and reports token balancedness.

Placement is not the final routing decision. When a logical expert has two replicas, the runtime still chooses which physical copy processes each token in the current batch.

A token-balancing router tries to make per-GPU token counts similar. An activation-balancing router tries to limit how many distinct expert replicas each GPU invokes.

Fanout's EPLB explainer covers replication and movement. The narrower issue here is the per-batch route after those replicas already exist.

Equal token load can double activations

Consider four GPUs and four logical experts, A through D. Each expert has two possible replicas, and the batch sends four tokens to each logical expert.

Suppose GPU 0 hosts A and B, GPU 1 hosts A and C, GPU 2 hosts B and D, and GPU 3 hosts C and D.

A uniform split sends two A tokens and two B tokens to GPU 0. GPU 1 receives two A and two C, GPU 2 receives two B and two D, and GPU 3 receives two C and two D.

Every GPU processes four tokens and activates two expert replicas. The route is perfectly token balanced.

Now send all four A tokens to GPU 0, all four C tokens to GPU 1, all four B tokens to GPU 2, and all four D tokens to GPU 3.

Every GPU still processes four tokens, but each activates one expert. Token balance is unchanged while the maximum activation count falls from two to one.

This constructed example is the cleanest form of the conflict. A token-only objective is indifferent between the routes, but a memory-bound kernel is not.

Why decode prices a weight load first

An MoE grouped GEMM groups the tokens assigned to each local expert. Before that expert can process tokens, its weights must reach the execution path through HBM and cache.

For a small decode batch, one token and several tokens may both fit inside the same kernel tile. The expert's first token pays most of the weight-read cost, while extra tokens reuse those weights.

If one expert's weight payload is S bytes and usable bandwidth is B bytes per second, the weight-streaming floor is roughly S divided by B before compute and communication overhead.

Activating two experts can approach twice that floor even when their combined token count matches one expert's token count. Counting only tokens misses the fixed cost per active expert.

The METRO paper measured a strong correlation between activated experts and vLLM decode latency.

Its Qwen3 example found that 1.5 times expert replication raised activated experts per decode batch by nearly 30 percent under EPLB. Decode latency rose 14 percent and total token throughput fell 10 percent.

Those numbers describe the paper's 8-GPU A100 setup, model, dataset, and routing implementation. They are a mechanism test, not a promise for another deployment.

METRO routes a live expert as one unit

METRO stands for Minimum Expert Token ROuting. It keeps expert placement and replication fixed, then changes how tokens select among the available replicas during memory-bound serving.

The paper observes that an optimal activation-only solution can send all tokens for one live expert to one replica. Splitting that expert cannot lower the number of active replicas.

The exact assignment becomes a restricted makespan problem over live experts, candidate GPUs, and a maximum activation count. The paper found the exact solver too expensive for the serving path.

Its greedy method processes active experts and selects the candidate GPU with the fewest activations. GPU counters and ordered locks keep concurrent assignments consistent.

METRO also gathers global tokens before top-k routing so each rank sees the expert demand for the whole batch. It then combines outputs with an all-to-all step.

In the paper's batch-32 tests, METRO's maximum activated-expert count was within 10.9 percent of the exact optimum and up to 42.3 percent below EPLB routing.

Current upstream vLLM documentation does not say that enabling EPLB selects METRO. Treat METRO as a paper implementation unless the deployed serving stack exposes and verifies that router.

Prefill can prefer token balance

Prefill processes many prompt tokens in parallel. Once each active expert has enough work, matrix multiplication leaves the flat weight-streaming region and compute grows with token count.

In that regime, concentrating all tokens for one hot expert on one GPU can create a long straggler. Splitting the tokens across replicas may cost another activation but shorten the critical path.

The METRO study found EPLB reduced prefill latency by 17 percent at 32 requests per GPU in its Qwen3 setup. Small prefill batches of 8 and 16 requests per GPU remained memory bound and did not gain.

This is why "balance activated experts" is not a universal replacement for token balance. It targets a regime where the fixed expert cost dominates the marginal token cost.

Disaggregated serving can select different routing policies for prefill and decode pools. A colocated engine needs phase-aware measurements because mixed batches combine both kinds of work.

Fanout's prefill and decode guide develops the difference in arithmetic intensity and latency goals.

Hot and cold experts can mix in one batch

A batch can contain a few hot experts with enough tokens to be compute sensitive and many cold experts that pay mostly for their first activation.

The recent TEMPO paper measured this mixture in recorded decode traces. It argues that neither token count nor activation count prices every expert correctly.

Its cost model uses a fixed activation term plus a token-dependent term and tile boundaries. The routing objective then minimizes the slowest GPU under the measured grouped-GEMM curve.

That refinement matters near the crossover. Whole-expert routing is efficient for cold experts, while a very hot expert may need a split to keep one GPU from becoming the straggler.

TEMPO reported that 92 to 100 percent of its recorded batches contained both regimes. It also reported a negative control where communication dominated and the mechanism added cost without a gain.

Activated-expert count is a better decode proxy than tokens when the kernel has a large flat cost. A calibrated time model is better when the batch spans both regimes.

Replication creates both opportunity and risk

Without a redundant copy, a logical expert has one destination. The router cannot trade token balance against activation count for that expert.

Replication adds choices. It can absorb a hot expert's load, but a naive uniform split can activate every copy on every batch even when one copy could handle the work.

The METRO paper found its advantage grew with replication because EPLB activated more replicas while METRO could keep one active copy per live expert.

Replication also consumes weight memory. Fanout's EPLB window memory guide separates that weight cost from the smaller load-statistics tensors.

Sweep replication ratio and routing policy independently. If both change together, lower latency may come from better placement, more memory bandwidth, or a different activation pattern.

Record logical experts requested, physical replicas available, physical replicas activated, tokens per activated replica, and the maximum per-GPU kernel time.

Test the proxy against the kernel

First profile grouped GEMM with the deployed expert shape, dtype, GPU, and kernel. Sweep both tokens per expert and active experts per GPU.

Find the token count where another token begins to cost more than another activation. Repeat the sweep with warm-cache and weight-rotation controls so accidental L2 reuse does not define the result.

Replay real router traces through at least three policies: current EPLB routing, whole-expert activation balance, and token balance. Keep expert placement fixed for this comparison.

Measure per-layer makespan, time per output token, prefill time, all-to-all time, and total throughput. Report p95 and p99 as well as means because one overloaded GPU gates the EP group.

Then repeat with placement updates enabled. Routing repairs a batch-level split; EPLB placement repairs where replicas live over a longer load window.

Do not optimize the logged balancedness ratio alone. A higher token-balancedness score can accompany more activated experts and slower decode.

Choose the objective by regime

Use token balance when expert work is compute bound and tokens per expert explain the kernel time. Large prefill batches commonly move in this direction.

Use activated-expert balance when small decode batches sit on the weight-streaming floor. In that case, keep each live expert on one replica unless a measured bottleneck justifies a split.

Use a calibrated makespan model when hot and cold experts share a batch or communication changes the winner. Multi-node topology can outweigh either local GPU proxy.

Keep the service SLO as the final test. A lower activation count is useful only when it reduces time per output token or admits more SLO-qualified throughput.

EPLB activated experts vs token balance is not a debate about one permanent metric. It is a routing choice that should follow the measured cost curve of the exact MoE kernel and batch phase.