Troubleshooting

Most surprising load-test results are the generator, the pacing, or the cardinality. Here is how to tell which.

Reading the numbers

http_reqs far below what you expected

Usually the generator, not the target. Two things to check first:

betweenRequests. It defaults to one second, so fifty virtual users with a single-request scenario produce roughly fifty requests a second, not thousands. That is deliberate — see betweenRequests — and --between-requests 0 removes it.

Where the VUs are spending their time. Add up the think steps in the scenario. A scenario with 4s of think time and one request runs one iteration every five seconds per user, whatever the VU count says.

Then check whether maxIterationRate is throttling.

duration climbing while waiting stays flat

Not the server. http_req_duration is the whole request; http_req_waiting is time to first byte — the server's own think time. When the first rises and the second does not, the extra time is going into connection setup, TLS, or a saturated generator.

Check http_req_connecting, and whether maxIdleConnsPerHost is too low.

A p99 far above the p95, with a flat max

The max is probably the histogram ceiling. Observations past 60s are recorded at 60s, so a flat max at that value means requests are hitting their timeout and their real durations are unknown.

Raise the timeout if you want to know how slow they actually are — or accept that "timed out" is the answer.

iteration_duration much larger than the sum of the requests

Time is going somewhere other than HTTP: scenario logic, JSON decoding, or contention in your own code. Think time and request pacing are both already excluded from this metric, so they are not the explanation.

A Go scenario doing expensive work per iteration is also competing with the generator for CPU.

Throughput flat while the VU count keeps climbing

The generator is saturated, or the service is. Distinguish them by looking at the load host's CPU and at http_req_connecting: if both are rising, it is you. Add a machine — see Distributed runs.

Percentiles that read slightly above the maximum

Expected, not a bug. max is tracked exactly; a percentile is the upper bound of the histogram bucket it falls in, and at 1% precision that can land a shade above the largest value observed. See Aggregation.

Numbers that disagree with the thresholds

You are almost certainly folding series rather than reading totals. Means need re-weighting by count and percentiles need the distributions, not their summaries. Use totals.

Warnings that mean the run is not trustworthy

"samples were dropped by nodes that hit their series cap"

warning: 1,284 samples were dropped by nodes that hit their series cap;
       a high-cardinality tag is the usual cause

A label somewhere has unbounded values. The usual suspects:

• a vu.Tag carrying a user id, a timestamp or a request id
• a step name set to something containing an id
• a derived request name whose path segments did not collapse

Every distinct label combination is a separate time series held for the length of the run. Find the runaway tag before trusting any number from this run — the reported figures understate reality.

"metric batches arrived too late to be counted"

warning: 12 metric batches arrived too late to be counted; check clock skew between hosts

Clock skew between load hosts. A few hundred milliseconds is absorbed; seconds are not. Run NTP everywhere.

The failed-requests list says it is partial

Distinct failure kinds are capped, per node and again on the coordinator. Past the cap new kinds are dropped and counted. A run producing that many distinct failures usually has a broken environment rather than an interesting result.

Things that fail before the run starts

"no agents"

run exits 1. Either no agent connected, or fewer than --agents did within --wait-agents. Check that the agents can reach the coordinator's --listen address — they dial out, so the reachability requirement is one-directional.

An unknown field

Rejected, not ignored. That is deliberate: a misspelled key would otherwise produce a run that appears to work and quietly measures something other than what you asked for.

loadwave validate test.yaml

A scenario name that does not exist

A YAML entry with only a name refers to a scenario compiled into the binary. If you are running plain loadwave against a config that names Go scenarios, they are not there — build the SDK binary and run that instead. See YAML or Go?.

The dashboard says it was not built into this binary

LoadWave: the dashboard has not been built into this binary.

You built with go build rather than make build. The REST API still works. To get the UI:

make ui && go build -o bin/loadwave ./cmd/loadwave

or without make:

npm --prefix web ci
npm --prefix web run build

then rebuild.

The live stream will not connect through a proxy

The WebSocket origin check is strict. Tell the server the origin it will be reached on:

loadwave serve --allowed-origin https://loadwave.internal.example.com

Turning up the detail

loadwave run test.yaml --log-level debug
loadwave run test.yaml --log-format json    # when something is collecting it

The event log in the dashboard and the HTML report carries the same story at a higher altitude: agents joining and leaving, quotas recomputed, workers dying, thresholds aborting. When a result looks strange, read it before re-running.

Next