Dynamic speculative decoding batch size thresholds

Dynamic speculative decoding batch size thresholds choose a draft length K from the live decode batch. Small batches can profit from more proposed tokens, while large batches can make parallel verification too expensive.

The threshold changes K. In vLLM it can also change CUDA graph mode, drafter work, tensor shapes, and which runtime combinations are supported.

Tune each K band against a true no-speculation baseline. A schedule that selects K = 0 can still cost more than running an engine with speculation disabled.

Dynamic speculative decoding batch size thresholds

Speculative decoding uses a smaller drafter to propose K tokens. The target model verifies those positions together and accepts the matching prefix without changing the target distribution.

Fanout's speculative decoding guide covers the draft, verify, accept, and reject loop. Dynamic scheduling adds one control: K changes with the active batch.

The current vLLM dynamic speculative decoding guide describes each band as start batch, end batch, and K. Both batch limits are inclusive.

A schedule might use K = 3 for batch 1 through 64, K = 1 for 65 through 128, and K = 0 for 129 through 512.

vLLM's explanation uses B times K as the effective verification load. Raising either the live batch B or draft length K can move target verification from spare compute capacity into a slower regime.

The purpose of the schedule is to reduce K before that crossover. The configuration does not discover the crossover for you.

A threshold should come from a measured K sweep

Run the same workload at fixed batch buckets with K equal to 0, 1, 2, and the largest supported draft depth. Keep prompt length, output length, prefix-cache state, sampling, and request order fixed.

At batch 32, K = 4 creates a B times K value of 128 under the documentation's convention. At batch 128, K = 4 raises it to 512.

If target verification time bends upward near 256 positions, a first schedule might reduce K before B times K reaches that region. This is a testable starting point, not a portable constant.

Accepted length must be in the same sweep. A cheap verification of four rejected drafts can still lose to one normal target step.

The current vLLM acceptance metrics expose proposed drafts, accepted drafts, verification steps, and an accepted-length histogram.

Mean acceptance length equals 1 plus accepted draft tokens divided by verification steps. Compare that progress with target verification time and drafter time for each batch bucket.

Choose the largest K that improves time per output token or SLO-qualified throughput in that bucket. Do not choose it from acceptance rate alone.

K equal to zero is not a no-spec engine

A dynamic band with K = 0 asks the configured speculative engine to produce no draft tokens for that step. It does not necessarily remove all speculative machinery.

Current vLLM drafter source keeps the drafter KV cache synchronized before returning an empty draft tensor when dynamic scheduling selects zero.

Startup configuration, buffers, scheduling checks, and graph decisions can also remain different from an engine launched without speculative decoding.

A recent vLLM performance report found a baseline tax across dynamic-schedule arms, including configurations that lowered K for much of the workload.

The report proposes separating four costs: the PIECEWISE graph downgrade, drafter work on K = 0 steps, speculative scheduling overhead, and behavior while a batch crosses a tier.

These are issue measurements from named revisions and hardware, not established costs for every release. They show why the correct control is a separate server launched with no speculative configuration.

Compare three arms at every high-batch band: static K, dynamic K = 0, and no-speculation. Only the third arm reveals the residual cost of carrying the dynamic path.

CUDA graph mode can move the whole curve

Dynamic K changes target verification length at runtime. Different verification lengths create different shapes for graph capture and replay.

The vLLM configuration source downgrades Model Runner V1 from full CUDA graphs to PIECEWISE when dynamic speculative decoding is active.

The documentation says full graphs require Model Runner V2 for this feature. It also says the schedule has been tested with EAGLE, EAGLE-3, and DFlash, while other methods may not work unchanged.

One concurrency regression report measured 59.6 tokens per second with static K = 2 and 50.0 after enabling a dynamic schedule in a single stream.

The same report observed a much larger collapse under eight concurrent requests: about 232 aggregate tokens per second became 24 to 157, and a 6.3-second run took 40 to 60 seconds.

The reporter linked the modest single-stream loss to the documented graph downgrade. The cause of the larger threshold-crossing collapse remained an open performance issue.

Do not attribute every regression to K selection. Record the resolved graph mode at startup and benchmark static K under that same mode before comparing schedule bands.

Boundary crossings can cost more than either steady state

A production batch does not stay at 64 or 65. Requests finish and arrive, so the scheduler can cross a threshold repeatedly during one measurement window.

At a boundary, K changes the query length and the amount of drafter and target work. The next step may use another captured shape or compile path.

A steady-state benchmark at batch 64 and another at 65 can miss this transition cost. Add a workload that oscillates around every configured boundary.

Record active sequences, selected K, target input tokens, drafted tokens, accepted tokens, graph mode, step latency, and any compile or capture event for each engine step.

Then split the result into stable-band steps and crossing steps. If only crossings stall, moving the threshold may relocate the problem rather than fix it.

The continuous batching guide explains why active batch size changes token by token. A request-level concurrency average hides the exact B used by the schedule.

For an operational test, replay a burst that fills above the boundary and drains below it. Repeat enough times to separate a first-use graph cost from recurring shape churn.

Data parallel and model runner limits are hard gates

Dynamic batch-size scheduling is not compatible with vLLM data parallelism in the current documentation. Independent ranks can choose different K values, diverge at collectives, and deadlock.

The current configuration therefore disables the batch schedule when data-parallel size is greater than one and falls back to the static speculative-token count.

Check the startup log rather than assuming the JSON was honored. A syntactically accepted schedule may not be the effective runtime policy.

Model Runner V2 is also not a blanket fix. A separate MTP graph-capture issue reports an assertion failure with dynamic K and full graph capture on one H100 setup.

CPU support has had its own open failure around changing K. Backend, drafter method, model runner, graph mode, and parallel layout form one compatibility matrix.

Pin the serving revision and test the exact matrix before rollout. Main-branch documentation and issue status can change faster than a production image.

Build the schedule from service objectives

Start with a fixed-K sweep and a no-spec server. For each batch bucket, collect time per output token, inter-token latency, target and drafter GPU time, mean acceptance length, and aggregate throughput.

Convert the winning fixed points into broad bands. Leave room around noisy crossovers instead of placing a threshold on one benchmark sample.

Replay variable arrivals that cross each boundary. Report stable-band performance separately from transition steps and from the first warm-up pass.

Test output equivalence too. The batch speculative decoding study shows that ragged accepted lengths can corrupt positions, masks, or KV state in incorrect implementations.

Finally, compare the schedule with the service metric that matters. Fanout's goodput guide shows why more aggregate tokens can coexist with worse SLO attainment.

Dynamic speculative decoding batch size thresholds are useful when they approximate a measured K curve without adding a larger transition or runtime-mode cost. Treat K = 0 as one schedule arm, not proof that speculation disappeared.