Tail Latency

Keep the useful bits, human.

Beginner field notes

BeginnerIntermediateAdvancedExpert

Tail latency concerns the higher-duration end of an observation distribution: responses taking longer than much of the measured set. An average alone does not show that end.
First name the operation, timing boundary and unit. This lesson measures a client’s label-read from sending to observing its reply, not later business completion.
A percentile is a value threshold under a specified calculation. It is not a fixed list of request identities or a prediction of the next request.

See a worked example

Same mean, different observed upper end

Sample only. Nothing here runs on your computer.

Example input · do not paste

OFFLINE comparison of two complete invented Lantern Archive batches. Durations are ms from client send to that client observing the response, for the same fictional operation and boundary.
Batch A observations: [10, 1000, 10, 10, 10, 10, 10, 10, 10, 10].
Batch B observations: [109, 109, 109, 109, 109, 109, 109, 109, 109, 109].
No additional observations, failures, timeout policy, concurrency, fan-out model or real collection window is supplied.
Mean = sum / count, with no rounding up. For both batches use only this nearest-rank convention: sort ascending with ties kept; rank = ceil(p * n), 1-based, 0 < p <= 1; no interpolation.
Compare n, total, mean, p50, p90, p99. Does the shared mean establish the same tail or a reliable service population p99?

Example output · do not paste

Batch A sorted: [10, 10, 10, 10, 10, 10, 10, 10, 10, 1000] ms.
A: n = 10, total 1090 ms, mean 109 ms; rank 5 / p50 = 10 ms, rank 9 / p90 = 10 ms, rank 10 / p99 = 1000 ms.
Batch B sorted: [109, 109, 109, 109, 109, 109, 109, 109, 109, 109] ms.
B: n = 10, total 1090 ms, mean 109 ms; rank 5 / p50 = 109 ms, rank 9 / p90 = 109 ms, rank 10 / p99 = 109 ms.
Same means: true. Same observed p99: false. Observed p99 difference = 1000 - 109 = 891 ms.
A’s one 1000 ms observation is 10% of A. Each p99 is the batch maximum under the chosen rule, not reliable population p99 or service-level objective (SLO) evidence.
No causal diagnosis, fan-out probability, future latency or success of downstream work follows.

Both complete batches have the same arithmetic total and count, while their observed upper ends differ. Retain the full sample, timing scope and method when interpreting a percentile. This comparison does not average per-instance percentiles, approximate a population tail, or prescribe a real-system change.

Visual

Two complete ten-observation batches have the same exact 109 ms mean but different observed nearest-rank p99 values: 1000 ms and 109 ms. Tiny-batch maxima do not establish population-tail behavior.

The reviewed description and complete diagram source are available here.

Read the complete diagram source

This is the source for the reviewed diagram, not a command to run.

flowchart LR
  A["Batch A: nine 10 ms + one 1000 ms"] --> M["Mean 109 ms"]
  B["Batch B: ten 109 ms"] --> M
  A --> PA["Observed p99 = 1000 ms"]
  B --> PB["Observed p99 = 109 ms"]
  PA --> S["Same chosen nearest-rank rule; not population / performance-target evidence"]
  PB --> S

Sources and full explanation

Notes sources · Beginner

AI source review · 2026-10-17

Reviewed explanation

Tail latency concerns the higher-duration end of a latency distribution: the observations that took longer than much of the stated set. An average does not tell you where those slow observations lie. Name what you measured first. This lesson times a fictional client’s label-read request from client send to that client observing the response, in milliseconds. It does not infer successful later work from a quick response, and it does not compare unlike timing boundaries.

Compare two complete paper batches under that same boundary. Batch A contains nine 10 ms observations and one 1000 ms observation. Batch B contains ten 109 ms observations. Each has count 10 and total 1090 ms, so each mean is exactly 109 ms. Their shapes differ: no observation in A equals the mean, while every observation in B equals it. Knowing only the equal means would hide A’s 1000 ms response and B’s uniformly 109 ms responses.

Use the foundation’s explicit nearest-rank rule for both batches: keep every observation, sort ascending and select 1-based ceil(p * n), with 0 < p <= 1 and no interpolation. In A, p50 and p90 are 10 ms but p99 is 1000 ms. In B, p50, p90 and p99 are all 109 ms. Percentiles name a threshold under a specified method, not a fixed set of request identities or a guarantee that exactly the named percentage is strictly below it. With just ten observations, ceil(0.99 * 10) = 10 selects the maximum; one long observation is 10% of A, not 1%.

The original Tail at Scale paper explains why component variability can matter to a larger distributed request. If a request must collect several required component results, one slow component can delay the combined answer. The actual dependency, waiting policy and correlation between component delays matter. Our two batches contain no fan-out topology, joint probability model or causal measurement, so they cannot supply a fan-out probability, identify a bottleneck or prove that a particular mitigation will help.

Real monitoring adds further interpretation choices. State the collection window and which observations or outcome classes are included, and check how missing, timed-out or canceled requests are represented rather than silently treating them as successful fast responses. Prometheus documents different quantile behavior for histograms and summaries, including estimation error and collection configuration. Averages of separately reported instance p99 values do not in general recover the overall p99. This exercise has the individual values and one declared convention; it is not a Prometheus estimator simulation.

These tiny invented batches demonstrate that equal averages can conceal different slow experiences. They do not reliably estimate a larger population’s 99th percentile, prove a user-facing service-level objective (SLO), establish that 1000 ms is an acceptable threshold or predict the next request. Queueing and back-pressure are related topics, but duration summaries alone do not establish either as the cause. Investigating or changing a real system requires its own evidence and constraints; this lesson runs no requests, retries or live performance experiment.

NIST / SEMATECH e-Handbook: 7.2.6.2 Percentiles
Explains sorted observations, ranks and the existence of different sample-percentile estimation / interpolation methods. Our explicitly defined ceil(p * n) nearest-rank rule is a chosen no-interpolation convention, not the NIST default or a claim that all tools use it.
Prometheus documentation: Histograms and summaries
Distinguishes count / sum averages from quantiles of observed durations, with measurement-window, histogram / summary estimation and aggregation qualifications. The paper exercise uses all supplied individual observations, not a PromQL query or bucket-based estimate.
Jeffrey Dean and Luiz André Barroso: The Tail at Scale (CACM, 2013)
The original paper discusses component latency variability and its possible amplification when large distributed requests depend on multiple component results. This lesson makes no numerical fan-out probability, independence, cause, universal mitigation or real-system performance claim.

AI source comparison recorded 2026-10-17. Human learner validation is pending.

Quick quiz · one question

Compare batches that share an average

Sample scenario. Two separate, complete invented ten-observation batches for the same fictional operation/boundary. No causal, fan-out, future, success or population claim. Integer ms observations; keep ties; mean=sum/count; chosen one-based nearest rank ceil((percent/100) × count), no interpolation.

Observed measurement boundary: All listed values are supplied fictional integer milliseconds from client send to client-observed response. No requests, collection, hidden observations, timeout policy, causal model, population estimate or SLO evidence.
Chosen percentile convention: Sort ascending with ties retained. For a requested percentage, round up (percentage / 100) × observation count, then select that one-based position. No interpolation or averaging of adjacent positions. Other conventions, including a default median, can differ.
Requested percentages: p50, p90, p99
Observed batch A: 10, 1000, 10, 10, 10, 10, 10, 10, 10, 10 ms. Arithmetic mean is the sum divided by count, without rounding it upward.
Observed batch B: 109, 109, 109, 109, 109, 109, 109, 109, 109, 109 ms. Arithmetic mean is the sum divided by count, without rounding it upward.

Both complete batches have mean 109 ms. Under the stated nearest-rank convention, do their observed p50, p90 and p99 also match?

Answers stay on this device.

Read the worked answer

Identical averages do not identify a distribution. The supplied batches have different observed typical and slow values under the same stated convention. Neither tiny batch establishes a service population p99, future behavior, a cause, a fan-out probability or a service-level objective.

Observed batch A: Sorted observations: 10, 10, 10, 10, 10, 10, 10, 10, 10, 1000 ms. Count 10; sum 1090 ms; arithmetic mean 109 ms. These are supplied fictional observations, not a future population or SLO estimate.
A: observed p50: Round up (50/100) × 10 to one-based rank 5; the observation there is 10 ms. Ties stay in the list; no interpolation or averaging of adjacent ranks.
A: observed p90: Round up (90/100) × 10 to one-based rank 9; the observation there is 10 ms. Ties stay in the list; no interpolation or averaging of adjacent ranks.
A: observed p99: Round up (99/100) × 10 to one-based rank 10; the observation there is 1000 ms. Ties stay in the list; no interpolation or averaging of adjacent ranks.
Observed batch B: Sorted observations: 109, 109, 109, 109, 109, 109, 109, 109, 109, 109 ms. Count 10; sum 1090 ms; arithmetic mean 109 ms. These are supplied fictional observations, not a future population or SLO estimate.
B: observed p50: Round up (50/100) × 10 to one-based rank 5; the observation there is 109 ms. Ties stay in the list; no interpolation or averaging of adjacent ranks.
B: observed p90: Round up (90/100) × 10 to one-based rank 9; the observation there is 109 ms. Ties stay in the list; no interpolation or averaging of adjacent ranks.
B: observed p99: Round up (99/100) × 10 to one-based rank 10; the observation there is 109 ms. Ties stay in the list; no interpolation or averaging of adjacent ranks.

Enable JavaScript to check a prediction and save a local practice attempt.

Sources and exercise scope

NIST / SEMATECH e-Handbook: 7.2.6.2 Percentiles
Explains sorted observations, ranks and the existence of different sample-percentile estimation / interpolation methods. Our explicitly defined ceil(p * n) nearest-rank rule is a chosen no-interpolation convention, not the NIST default or a claim that all tools use it.
Prometheus documentation: Histograms and summaries
Distinguishes count / sum averages from quantiles of observed durations, with measurement-window, histogram / summary estimation and aggregation qualifications. The paper exercise uses all supplied individual observations, not a PromQL query or bucket-based estimate.

Original synthetic exercise. AI comparison against the reviewed lesson recorded on 2026-10-17; the references above state their scope. Human learner validation has not been run. A result here records this attempt; it does not establish mastery.

Save or adjust this lesson

Reviewed videos and Shorts

Videos · 16:9

Shorts · 9:16

Beginner field notes

Same mean, different observed upper end

Visual

Notes sources · Beginner

Reviewed explanation

Compare batches that share an average

Make this lesson yours

Understand what can change

File removal

Public access