NVFP4 KV cache prefill vs decode performance

NVFP4 KV cache prefill vs decode performance can move in opposite directions. A packed 4-bit cache reduces memory traffic, but the current attention path may pay to unpack, scale, and convert those values.

That trade can slow a cold causal prefill while speeding bandwidth-bound decode. It can also improve prefill indirectly when greater cache capacity prevents eviction and turns recomputation into a prefix hit.

Benchmark those effects separately. A lower time to first token under reuse does not prove the raw NVFP4 prefill kernel is faster than FP8.

NVFP4 KV cache prefill vs decode performance

Prefill processes many prompt tokens in parallel and writes their key and value vectors into the cache. Decode reads the growing cache for one new token at a time and appends one new K/V entry.

The NVIDIA NVFP4 KV cache study describes a current path that dequantizes NVFP4 cache values before attention math.

The new token's K and V vectors are quantized before the server appends them. The cache saves bytes, but conversion and scale handling add work on both reads and writes.

Prefill is usually more compute-heavy because it exposes parallel work across the prompt. Decode is usually more sensitive to memory bandwidth because each sequential step reads prior cache state.

Fanout's prefill and decode disaggregation guide explains why the phases already prefer different scheduling and hardware. Cache quantization adds another phase-specific difference.

NVFP4 stores 4.5 bits per value before padding

Current vLLM NVFP4 cache code packs E2M1 values and stores one E4M3 scale for each group of 16 elements.

Sixteen 4-bit values use 64 bits. One 8-bit scale raises the group to 72 bits, or 4.5 bits per original value.

Relative to an 8-bit FP8 value, the ideal data-and-scale ratio is 4.5 divided by 8, which is 56.25 percent. The ideal capacity multiplier is 8 divided by 4.5, or about 1.78.

Actual pages can add padding, alignment, allocator slack, and backend-specific metadata. Do not advertise a 1.78 times concurrency gain until the resolved cache layout and workload produce it.

The same vLLM kernel quantizes BF16 or FP16 input, creates packed FP4 data, writes block scales, and maps the result into paged cache slots.

Those operations explain why cache bytes and cache-write time do not shrink by the same factor.

A worked per-token cache estimate

For ordinary multi-head KV storage, FP8 bytes per token equal 2 times L times H times D. L is attention layers, H is KV heads, D is head dimension, and two accounts for K and V.

Take L = 64, H = 8, and D = 128. FP8 uses 2 times 64 times 8 times 128 bytes, which is 131,072 bytes or 128 KiB per token before sharding and page overhead.

The ideal NVFP4 data-and-scale footprint is 56.25 percent of that value, or 73,728 bytes and 72 KiB per token.

At 131,072 cached tokens, those estimates become 16 GiB for FP8 and 9 GiB for NVFP4 per unsharded sequence. The seven-GiB difference can admit another request or retain an older prefix.

This example isolates cache format. Grouped query attention, tensor or context parallelism, hybrid layers, page padding, and replicated metadata change the device-local total.

Fanout's NVFP4 cache and weight guide keeps this growing runtime allocation separate from fixed model weights.

Cold prefill can lose on conversion cost

A current vLLM performance report compares matched profiler segments on an RTX PRO 5000 Blackwell GPU using FlashInfer 0.6.15.

Across 48 causal paged-prefill calls, the reported NVFP4 path took 145.81 ms while FP8 took 84.03 ms. That is a 1.74 times regression for that SM120 software and hardware stack.

The report attributes the gap to a BF16-query and packed-NVFP4-KV path. K values must be unpacked, converted, and scaled before BF16-oriented QK computation.

The CUTLASS GEMM totals in the matched segment were similar. Most extra GPU time appeared inside the paged-prefill attention kernel rather than the model's weight or MoE kernels.

This is one open issue, not a universal NVFP4 ratio. SM100, SM120, FlashInfer versions, query dtype, head layout, and attention kernels can change the result.

It still exposes the correct diagnostic. Compare the attention kernel with identical cached tokens before attributing an end-to-end TTFT change to the format.

The FlashInfer NVFP4 merge added both prefill and decode kernels with per-block FP8 scales. Support does not imply equal maturity across every path.

Decode can win on fewer bytes read

The same vLLM report measured single-token paged attention at about 9.63 microseconds per call for NVFP4 and 12.35 microseconds for FP8.

That makes NVFP4 about 22 percent faster in the reported decode workload. Decode repeatedly scans prior cache state, so the smaller representation can reduce the bandwidth term enough to pay for conversion.

Cache append moved the other way. The NVFP4 append kernel took about 8.39 microseconds per call versus 3.01 microseconds for FP8, roughly 2.8 times slower.

Append quantizes the new vectors, chooses block scales, packs values, and writes the slot. The report says its cumulative append cost was about seven milliseconds and was not the end-to-end bottleneck.

Do not infer decode speed from append speed alone. Attention reads a long history while append writes one new token, so their byte counts and repetition differ.

Fanout's KV quantization slowdown guide gives the broader rule: compression helps only when avoided memory work exceeds quantize and dequantize overhead.

Prefix hits can reverse the prefill result

The NVIDIA study reports up to three times lower TTFT and a 20 percent higher cache-hit rate for NVFP4 than FP8 on Qwen3-Coder-480B-A35B.

That result came from effective cache capacity. A smaller representation kept more previously computed context resident, so requests skipped prefill work that the FP8 setup had to recompute.

The benefit narrows once both formats have enough memory to retain the useful working set. At that point, there are fewer FP8 evictions for NVFP4 to avoid.

This creates two valid but different prefill statements. On a cold matched kernel, NVFP4 may be slower. On a capacity-bound reused workload, it may deliver lower TTFT by producing more hits.

Record cached prompt tokens beside TTFT. Without that denominator, a cache-capacity win can be mistaken for a faster arithmetic path.

Also separate lookup hits from useful reuse. A hit that saves only a short prefix may not offset NVFP4 conversion across the rest of a long prompt.

Capacity can avoid preemption without winning latency

The vLLM report also ran 30 concurrent requests. NVFP4 completed all 30 without preemption and peaked at 68.6 percent cache use.

FP8 completed the same requests with six preemptions, a peak of 24 running requests, and 97.6 percent cache use.

Even so, wall time was 102.3 seconds for NVFP4 and 65.4 seconds for FP8. In that workload, the causal-prefill regression outweighed the capacity benefit.

This is why completed requests, preemptions, throughput, TTFT, and inter-token latency all belong in the comparison. Capacity is valuable, but it is not latency by itself.

A different workload can reverse the outcome. Longer decode, more prefix reuse, or a tighter cache pool gives the reduced read volume and avoided eviction more chances to matter.

Benchmark the format by phase and traffic shape

Run one cold-prompt benchmark with prefix caching disabled or empty. Measure prefill attention time, cache append time, TTFT, and input tokens per second.

Run a decode-heavy benchmark with fixed prompt and output lengths. Measure single-token attention, append, inter-token latency, output tokens per second, and memory bandwidth.

Then run the production trace with prefix caching enabled. Hold request order constant and record cached tokens, evictions, preemptions, peak running requests, and cache utilization.

Sweep context length and concurrency until FP8 first reaches capacity. That region shows whether NVFP4 earns enough extra residency to offset its direct kernel costs.

Repeat on the exact GPU, FlashInfer build, vLLM revision, and attention backend intended for production. Current issue data is useful for a hypothesis, not a substitute for that matrix.

Choose NVFP4 when capacity or decode bandwidth is binding and measured quality stays acceptable. Keep FP8 when cold prefill dominates and the larger cache still fits the working set.

The phase split is the decision: prefill pays conversion and may gain hits, decode repeatedly saves bytes, and append pays the packing bill one token at a time.