Why averaging p99 across hosts is wrong

Use t-digest to merge latency distributions and estimate tail quantiles without retaining every observation.

T-digest quantile sketches

A service can have a comfortable p50 latency and an unpleasant p99. The median describes the middle request; the high percentile exposes slower requests that an average can hide. To query those percentiles across many hosts, we need more information than one average or p99 per host.

A t-digest keeps a compact, approximate description of a numerical distribution. We will look at what it stores, how local digests combine, and why a plausible percentile still needs an accuracy check.

Store weighted groups

For an exact reference, choose the nearest-rank convention: sort N observations and take the one-based position ceil(q × N) , where q is the requested fraction. At q = 0.99, that gives p99. Retaining and sorting every observation becomes costly for large streams and many dashboard queries. Exact frequency counts are another option when the value domain is small enough.

A t-digest compresses observations into centroids . Each stores a mean and a weight: the number of observations it represents. For example, grouping 10, 11 and 12 yields mean 11 and weight 3. That record preserves the group's mass and mean, but not its individual values.

The distinctive choice is where to spend detail. T-digest limits centroid weights more tightly near the distribution's ends and allows larger groups near the middle. This helps preserve tail resolution without retaining every observation. The scale function, insertion order and interpolation rules still affect the result.

Each host can build a local digest. A backend combines compatible weighted centroids, recompresses them and queries the combined distribution. The weights prevent a quiet host from receiving the same influence as a busy one. This merge does not recognize duplicated observations or overlapping uploads; wiki/mergeable-sketches-for-analytics the merge lesson covers that accounting separately.

What a query returns

A percentile query uses the ordered centroids and their cumulative weights, often interpolating between them. Its answer can lie between values that actually occurred. That is different from our nearest-rank reference, which always returns an observed value.

Check two kinds of error:

Value difference: how far the estimate is from the exact reference, in the measurement's units. Rank distance: how far the estimate lies from the requested fraction of the population.

For ties, use a rank interval: the fraction strictly below the estimate through the fraction at or below it. The rank distance is zero if q lies inside that interval; otherwise it is the distance to the closer endpoint. This lets an observed value represent a whole block of equal observations.

A small rank error can mean a large latency error in a sparse tail. A small latency difference can also cross a large repeated mass. Neither measure substitutes for the other.

Run one pinned implementation

The example uses Cam Davidson-Pilon's Python tdigest package, version 0.5.2.2. It is a reproducible implementation study, not a claim that every package called t-digest has the same behavior.