Aggregation
Why percentiles cannot be averaged, how HDR histograms merge losslessly, and the retention trade-offs behind the charts.
This is the part that is easy to get subtly, silently wrong, so it is worth understanding.
Percentiles cannot be averaged
The p99 of ten agents is not the average of ten p99s. That number does not mean anything. Getting it right means combining the underlying distributions, not their summaries.
LoadWave uses HDR histograms, which merge losslessly. Each node ships its whole distribution and the coordinator merges them, so a p99 across a fleet is the true p99.
Every node must use an identical bucket layout or the merge is meaningless, so the resolution is fixed run-wide and a node reporting a different one is rejected rather than accommodated.
The default resolution is 0.1ms to 60s at two significant figures — about 1% relative error, which is far finer than the noise in any real load test, and roughly 13KB per histogram.
Three consequences worth knowing
A percentile can read slightly above the maximum
max is tracked exactly; a percentile is the upper bound of the histogram bucket
it falls in. At 1% precision a p99 can therefore come out a shade above the
largest value actually observed.
It is a property of bucketed histograms, not a bug — and the alternative, keeping every sample so percentiles are exact, does not survive contact with millions of requests.
Values above the ceiling are clamped, not dropped
An observation past 60s is recorded at 60s. It understates the outlier, but a request that hit its timeout is nearly always the most interesting sample in the run, and discarding it would make the tail look better than it is.
This is why a p99 far above the p95 with a flat max usually means requests are hitting their timeout, and their real durations are unknown.
Batches are deltas, not totals
A node reports what happened in each interval, never a running total. A dropped batch then costs one interval of data instead of skewing every interval after it — which matters because a node reconnecting after a network blip is routine.
Time buckets are aligned to the wall clock
Every node cuts its buckets at the same instants, so the coordinator can merge them without interpolating and a p99 for 12:00:03 really is everyone's traffic in that second.
Clocks need to be roughly in step, which NTP handles; a grace period absorbs the rest. Batches arriving after it are dropped and counted, and the report warns about clock skew:
warning: 12 metric batches arrived too late to be counted; check clock skew between hosts
Two levels of retention
Keeping a full histogram per series per second for an hour would cost tens of gigabytes. So the coordinator keeps:
- Cumulative aggregates at full label cardinality and full histogram fidelity, for the whole run. This drives the endpoint table and every threshold.
- Time buckets at reduced dimensionality — per metric and per scenario — holding scalars plus four precomputed percentiles. This drives the live charts. The percentiles are computed from the merged histogram before it is released, so they are correct; the histogram itself is then discarded.
- Per-endpoint buckets holding a sum and a count only, and therefore an average. This is what the response-time chart plots one line from.
The last of those is deliberately average-only. A histogram per endpoint per second would cost gigabytes over an hour, where a sum and a count cost thirty-two bytes.
Percentiles per endpoint are still exact for the whole run — that is what the endpoint table shows — but they are not available second by second. If you need a per-endpoint tail over time, narrow the run rather than the chart.
Use the totals, not the per-series numbers
The API exposes both series — one entry per label combination — and totals, one
correctly merged aggregate per metric. Always use totals for a whole-run figure.
Folding series yourself produces plausible-looking numbers that disagree with the
thresholds, because means need re-weighting by count and percentiles need the
distributions.
The same applies to endpoints: those percentiles are recomputed from each
endpoint's merged distribution across all of its status codes. Taking the maximum
of the per-status percentiles — the obvious shortcut — reports the tail of whichever
status happened to be slowest, and the two diverge most exactly when an endpoint
starts failing.
Where the caps are
Two bounded resources, both capped in two places and both visibly lossy rather than silently so:
| Resource | Cap | Past the cap |
|---|---|---|
| Time series | 5,000 per node, and again on the coordinator | Observations for new series are dropped and counted; existing series keep recording. |
| Distinct failure kinds | Per node, and again on the coordinator | New kinds are dropped and counted; the dashboard and report say the list is partial. |
A bounded, visibly lossy run beats an out-of-memory kill. Both cases produce a warning in the report, and a run carrying one should have its cardinality fixed before its numbers are trusted.