Mamba state quantization for inference

Mamba state quantization compresses the persistent matrix that a recurrent layer reads and rewrites at every generated token. It is not another name for quantizing model weights, activations, or a Transformer KV cache.

The state stays fixed in size as context grows, but every active request needs its own copy. At high concurrency, state traffic can consume gigabytes and a visible share of decode time.

The difficult part is feedback. A rounding error written today becomes part of tomorrow's input, so a format that looks acceptable in one isolated write can still change a long generation.

Mamba state quantization changes persistent state

A softmax-attention layer appends keys and values for each processed token. A recurrent layer instead summarizes the prefix in a fixed-size state and updates that state after every token.

For Gated DeltaNet and Kimi Delta Attention, one head keeps a matrix with key dimension by value dimension. The query reads that matrix, while decay and a delta-rule correction write the next version.

The DAMP paper draws the operational distinction cleanly. Weights stay fixed, KV entries are written once and read later, and recurrent state is repeatedly read, transformed, quantized, and written back.

That lifecycle makes state precision a serving choice. It affects the memory pool per live sequence and the bandwidth of every decode step.

Fanout's Kimi K3 architecture guide explains how KDA and full attention coexist.

Only the attention layers pay context-proportional KV growth. The recurrent layers pay a fixed state cost per request.

Recurrent state is fixed per request, not free

DAMP reports that Qwen3.6-35B uses 30 GDN layers. At a decode batch size of 256, its FP32 recurrent states occupy 15 GB of GPU memory.

Dividing 15 GB by 256 gives about 60 MB of recurrent state per active request in that setup. The value does not grow with prompt length, but doubling the live batch approximately doubles this pool.

The same profile attributes 24.3% of decode latency to state updates. Each token requires the runtime to read the matrix, apply the recurrence, and write the new matrix.

This is why a fixed-size state can still be bandwidth bound. Fixed with respect to sequence length does not mean fixed for the server.

At batch 32, the simple proportional estimate is about 1.9 GB. At batch 96 it is about 5.6 GB. At batch 256 it reaches the measured 15 GB.

A hybrid deployment must budget both shapes. KV memory grows with attention layers, cached tokens, and concurrency. Recurrent-state memory grows with recurrent layers and concurrency, regardless of cached-token count.

The KV-cache memory formula covers the token-growing side of that budget.

Quantization error returns on the next token

Let the uncompressed update be S_t = A_t S_(t-1) + W_t. Here A_t contains decay and the key-directed correction, while W_t writes information from the current token.

If quantization injects residual R_t after the write, the stored state follows S_q,t = A_t S_q,(t-1) + W_t + R_t.

The error therefore follows Delta_t = A_t Delta_(t-1) + R_t. Unrolling it leaves the newest residual plus older residuals multiplied by every later transition.

A KV-cache value does not pass through this feedback loop. It is reconstructed when attention reads it, but that reconstructed value does not replace the stored entry for the next token.

Decay can suppress old recurrent-state error, but it does not suppress every channel equally. DAMP found stable differences in persistence across KDA key channels and across GDN heads.

This explains why one-step reconstruction error is an incomplete calibration target. The dangerous channel is both inaccurate when quantized and slow to forget that error.

Uniform eight-bit state can still fail reasoning

The DAMP experiments tested Qwen3.6-35B and Kimi-Linear-48B across math, general reasoning, and code tasks. Uniform FP8 and INT8 state storage did not preserve the FP32 result.

On Qwen3.6, FP32 state scored 85.46 on AIME 2026 and 86.95 on LiveCodeBench v6. Uniform FP8 state fell to 29.27 and 15.02. Uniform INT8 reached 18.48 and 32.23.

A Hadamard transform before INT8 improved those scores to 44.21 and 34.08, but a large gap remained. Four-bit formats nearly collapsed the tested math and code results.

The result is specific to those checkpoints, storage quantizers, and evaluation settings. It still rules out the shortcut that eight bits is automatically safe because the memory ratio looks modest.

The Nemotron 3 Super technical report found a subtler failure from FP16 state.

On an early checkpoint, FP16 state kept code pass rates near the FP32 baseline but increased LiveCodeBench completion tokens by 36.95%. With W8A8 weights and activations, the increase was 40.27%.

Accuracy alone would miss that behavior. Output length changes cost, latency, and user experience even when pass rate remains flat.

Mixed precision spends bits on persistent channels

DAMP first applies a Hadamard transform along each channel's value dimension. It then estimates low-precision reconstruction error and combines it with a decay-based persistence score.

The method protects the highest-risk key channels in FP16 and stores the remainder in INT8 with scale and zero-point metadata. Calibration fixes the channel selection before serving.

Its main layout protects 16 of 128 key channels. The low tier costs 9 bits per value including metadata, while the high tier costs 16 bits.

The average is 9 + (16 divided by 128) times (16 minus 9), or 9.875 bits per value. The paper reports that as 9.9 bits.

Relative to 32-bit state, 9.875 bits is a 69.1% storage reduction. Applying that ratio to the reported 15 GB pool gives about 4.6 GB, freeing roughly 10.4 GB at batch 256.

The packed layout matters as much as the arithmetic. Selected channels are permuted into contiguous regions so one fused kernel can load FP16 and INT8 tiers without irregular gathers on every token.

The kernel reconstructs the low tier, computes the recurrence in FP32, derives new quantization parameters, and writes the packed state back.

Mamba state quantization needs phase-aware benchmarks

DAMP measured recurrent-update speedups from 1.46 to 1.65 times for GDN and 1.81 to 2.01 times for KDA as batch size moved from 32 to 256.

The full-model gain was smaller because recurrent updates are only part of decoding. At batch 96, time per output token fell 5.3% for GDN and 4.8% for KDA. At batch 256, it fell 10.9% and 9.6%.

Those numbers show an amortization effect. Quantization and reconstruction cost stays visible at small batches, while saved state traffic matters more when many request states compete for memory bandwidth.

Prefill needs a separate result. DAMP computes recurrence in FP32 inside each chunk, then writes the chunk-final state into the packed cache. Decode repeatedly reads and rewrites that representation.

The SSDi8 paper makes a related point for Mamba-2's State Space Duality path. Keeping an INT8 representation across modules avoids conversions that erase the value of lower precision.

Do not infer end-to-end speed from bytes alone. Verify the actual kernel, the persistent storage type, and whether intermediate paths silently widen the state.

Test quality as a trajectory, not one answer

Begin with FP32 state and change no other precision. Record resident state bytes per request at several batch sizes, recurrent-kernel time, time per output token, and total admitted sequences.

Add a uniform FP16 or INT8 control before testing a mixed layout. That control reveals whether the gain comes from precision allocation or merely from narrower storage.

Use long generations from the real workload. Measure exact task quality, completion length, repetition, tool-call validity, and the first token position where a deterministic run diverges from the reference.

Repeat with short and long prompts. Recurrent-state size stays fixed, but prompt processing determines the state written before decode begins.

Pin the checkpoint, serving revision, state kernel, sampling settings, and calibration corpus. The DAMP calibration used 32 documents across math, biomedical text, code, and discussion data, not one convenient chat sample.

Mamba state quantization is worthwhile when concurrent state traffic is the measured bottleneck and the compressed trajectory preserves behavior. The deployment unit is the full recurrence, not an isolated tensor cast.