The system is slow. Which part is responsible?

An operator sees a sensor’s state change only after a second. The backend team reports a handling time of 30 ms. The gateway reports a quick send, and the browser receives its response without an error. All these observations can be true. They still do not explain where the rest of the second went.

In a distributed system, users wait for the outcome of the whole journey while teams usually measure their own sections. Difficult delays often occur between them: before transmission, before processing starts, or after data is stored but before the view changes. Speeding up a short endpoint can improve its chart without solving the user’s problem.

We will build a synthetic flow from a sensor through a gateway, queue and service to a dashboard. Three scenarios will have the same total duration but different causes. We will then compare the mean with the distribution’s tail and prepare a diagnostic procedure. All durations are explicit assumptions in a virtual-clock model, not measurements of a network, hardware or BBest24 deployment.

Define the start and finish first

In our task, the start is the event that the dashboard must represent. The finish is the visible change of the appropriate interface element. Writing to a database does not finish this measurement. Nor does the gateway receiving an acknowledgement. Those are useful intermediate points, but they answer different questions.

Write this distinction into the requirement. An API can respond within 100 ms while a user waits several seconds for periodic refresh. Requiring state to appear within 500 ms of the event for at least 99% of handled events is closer to user experience, but still needs load, connectivity conditions and error accounting to be specified.

Unsuccessful events must not be excluded simply because they have no completion time. If some updates disappear, the distribution of successful operations alone can improve while the product gets worse. Alongside duration, count started, completed, rejected and unfinished operations. A timeout is a separate outcome, not zero or an empty cell to discard.

For a physical event, also establish when the sensor can detect it. Slow sampling creates waiting before the first message exists. Our model allocates 20 ms to the initial stage; it does not prove that any sensor can detect a change that quickly. If timing starts at the gateway, call the result gateway-to-view latency rather than implying measurement from the physical event.

The same second can originate in three places

The baseline has six sequential stages: 20 ms for the initial sensing stage, 30 ms in the gateway buffer, 40 ms for transport, 20 ms waiting in the queue, 30 ms of service work and 60 ms until the dashboard visibly changes. Total duration is 200 ms. These stages are sequential and non-overlapping in this model.

Scenario A adds 800 ms to the gateway buffer. B adds the same delay to the queue. In C, data reaches the service normally but the interface waits another 800 ms. Every flow takes 1000 ms. Service work remains 30 ms in all three.

In A, a faster server does not remove waiting before transmission. Investigate batching rules, task scheduling and when the buffer drains. In B, the relevant signals include oldest-message age, arrival rate and processing-start rate. In C, inspect polling intervals, subscriptions, application-state updates and rendering.

This experiment also tests observability itself. If we know where the extra 800 ms was inserted but the report blames another component, instrumentation is incomplete or its interpretation is wrong. A trace that cannot distinguish these controlled situations is a poor basis for tuning the system.

The figures and arithmetic checks use the stage durations given above and the same synthetic set of 1000 results. Actual program execution delays are not presented as infrastructure measurements.

A trace needs identity, not just timestamps

A distributed trace connects operations belonging to one flow. OpenTelemetry describes a span as a unit of work with a start, finish and context. Connecting operations helps distinguish execution from waiting to start. Logs with roughly similar timestamps may be insufficient, particularly across many devices.

In our example, an event receives a stable logical identity. Each send attempt and handling attempt can have its own execution identity. Attempt number and retry reason remain visible. Three transmissions of the same state change are then distinguishable from three independent events or one unusually long attempt.

W3C Trace Context standardises context propagation through HTTP headers, including traceparent and tracestate. That does not mean a sensor pipeline automatically carries them through every protocol. At gateway, broker and binary-format boundaries, verify how the chosen implementation transfers and reconstructs the relationship.

For asynchronous work or operations covering multiple messages, span links described by OpenTelemetry can represent causal relationships that do not fit a simple parent-child tree. Preserve causality rather than stretching a single span across hours offline just to make everything resemble one request.

Boundaries that must leave evidence

The flow needs observations on both sides of important boundaries. The following table does not prescribe a telemetry supplier. It specifies the distinctions we must be able to make.

“Sent” must have one meaning. It might mean handing data to a local library, writing to an operating-system buffer or receiving a broker acknowledgement. If two teams use the word for different boundaries, subtracting their timestamps produces apparently precise but incomparable results.

Likewise, “displayed” is not synonymous with executing a function that updates application state. If the target is a visible effect, tie completion to the appropriate interface stage and verify it in a controlled trial. An inactive browser tab needs its own scenario rather than an unconditional assumption that it behaves exactly like an active operator screen.

Clocks can invent negative network latency

Suppose actual transport takes 40 ms but the sender’s clock is 80 ms ahead of the receiver’s. Subtracting the recorded timestamps gives minus 40 ms. We have not discovered negative latency. We have compared clocks with an offset.

For durations within one device, a monotonic clock is generally preferable because calendar-time corrections do not change the result. A counter from one boot must not be subtracted directly from a counter after restart. Boot identity and counter-wrap handling are needed when the counter has a limited range.

Across devices, clock synchronisation quality matters. If the measured difference is 50 ms and each timestamp may be wrong by up to 20 ms, a conservative interval for the real separation is 10 to 90 ms. This is not automatically a statistical confidence interval; it follows from assumed error bounds. Reporting 50.000 ms does not become more credible through additional digits.

Without justified synchronisation, we can still measure local buffer, queue and service durations and establish ordering through identities. We cannot confidently assign every unaccounted interval to the network. The report must retain an unmeasured segment or a bound requiring further instrumentation.

Do not add everything that has a duration

Adding stages works in our model because they are sequential and disjoint. Real traces can contain nested spans and parallel work. A service span lasting 120 ms and a child operation lasting 80 ms do not imply 200 ms of user waiting. Those 80 ms may already be inside the first interval.

For example, after 10 ms of preparation, start two parallel operations lasting 80 and 120 ms, then spend 10 ms combining their results. If both must finish, total duration is 10 + max(80, 120) + 10 = 140 ms, not 220 ms. The relevant quantity follows the critical path: dependencies determining the earliest possible finish.

The same caution applies to percentiles. Adding several services’ p99 values does not generally produce end-to-end p99. Their slowest samples may belong to different events, and dependencies between stages can change the tail. Compute the end-to-end percentile from whole-operation durations, preserving correlation rather than adding numbers from separate dashboards.

A 236 ms mean can conceal five seconds

Construct 1000 synthetic durations: 980 results at 200 ms, 15 at 1000 ms and 5 at 5000 ms. The mean is 236,000 / 1000 = 236 ms. Median and p95 are 200 ms. However, p99 is 1000 ms and p99.9 is 5000 ms.

We use nearest rank: for percentile p, select position ceil(p × n) after sorting, with positions starting at one. For p99, this is position 990. Other tools may interpolate or approximate quantiles, so the method and histogram boundaries belong in the report. With a small sample, a high percentile depends on very few observations.

In this dataset, 980 of 1000 events meet a 500 ms limit: 98%. The 99% target is missed even though the mean is far below the limit. The Google SRE monitoring chapter explains why averages hide slow tails and why retaining duration distributions is useful. Our numbers are an independent teaching example, not results from the service discussed there.

The chart shows sorted order, not event chronology. It cannot tell whether the five slow cases occurred during one incident or at five different installations. Diagnosis needs both views: a distribution showing the scale of the problem and context for individual slow operations.

Two fixes win by different measures

The first proposal reduces service work by 20 ms for every event. Mean duration falls from 236 to 216 ms, and p99 falls from 1000 to 980 ms. This is a real improvement in the model, but it still misses the 500 ms target for 99% of events. The longest cases take 4980 ms.

The second proposal removes the additional 800 ms from the fifteen queue cases, leaving the five 5000 ms cases unchanged. Mean duration becomes 224 ms, higher than after the first optimisation. Yet p99 drops to 200 ms: 995 of 1000 operations meet 500 ms. Against the stated objective, the second fix is more effective.

That does not mean the entire problem is solved. p99.9 remains 5000 ms. If those events concern one important installation or a critical operation, a fleet-wide aggregate may be too lenient. Choose metrics according to the consequences of waiting, not simply to obtain a convenient green indicator.

A real test must check whether the fix also changes rejection rate, load or resource consumption. Disabling a slow path can shorten successful response times while increasing errors. Compare the same boundaries, similar loads and the complete set of outcomes, not just a new p99 chart.

A queue is about rates, not only message counts

Queue length without context can mislead. A hundred short tasks may clear sooner than one very long task. Useful signals include oldest-item age, waiting-time distribution, active consumer count and arrival rate. In scenario B, delay occurs before the service begins its 30 ms of work, so profiling the CPU work alone cannot explain it.

If 50 events arrive per second and a timeout error causes each to be sent twice on average, the receiver sees 100 attempts per second. When four layers independently retry, the load model can become considerably more complicated. A simple example of four attempts for each of those hundred requests already gives 400 attempts per second. This is an assumed load scenario, not a universal retry multiplier.

Retries need a shared time budget, bounded attempts and a distinction between an event and its deliveries. A shorter timeout does not always reduce user waiting; it can create extra work that lengthens the queue. Investigate whether rising latency preceded retries or whether retries began after a configuration change and worsened the situation themselves.

The load generator can hide overload

Imagine a test that sends the next event only after the previous one finishes. When the system slows, the generator slows too. It reduces load precisely when a queue should be growing. That may represent one operator waiting for a result, but it does not reproduce a sensor fleet sending independently of server speed.

If the scenario requires 50 new events per second, record both the planned schedule and actual generation times. Insufficient generator capacity must not look like a lack of device traffic. Start delays and missed emissions belong in the test report. Otherwise, the trial may preserve low response times by silently failing to deliver part of the intended workload.

Traffic shape matters equally. A steady 50 events per second is different from a batch of 3000 once a minute, despite the same average rate. The latter may occur when devices share a schedule or reconnect after an outage. Test both if they are within operating conditions. A steady-load trial does not establish readiness for bursts.

Before comparing releases, preserve initial state as well: an empty or backlogged queue, warm or cold connections and the number of active consumers. If one release starts with a prepared cache while the other rebuilds it, the difference may not arise from the proposed fix. Those variants are valuable but should be named separately. A performance test must reproduce a specified user situation rather than produce one number detached from its conditions.

Trace sampling must not hide what we are investigating

Recording every operation in full may be too expensive. A decision to retain a trace at its start is simple but cannot yet know whether that operation will become unusually slow. OpenTelemetry distinguishes head and tail sampling. Tail sampling can use information about the completed path but requires resources and proper collection of relevant spans.

Do not estimate the general traffic distribution solely from a collection deliberately retaining slow traces. Such a set is useful for diagnosis but selected by design. Population metrics and detailed examples need clearly documented origins. Missing slow traces may also indicate an overloaded collector or broken context propagation rather than absence of a problem.

Minimum diagnostic context includes device and service versions, connection mode, operation outcome, attempt number and a waiting reason where known. It does not automatically require complete message payloads or customer data. Trace identities are not credentials, and the W3C specification prohibits personal or sensitive information in context fields intended for correlation.

A procedure from complaint to verified improvement

Start with one user-visible event and define the beginning and end. Then collect identities and intermediate durations, marking local clocks and uncertain differences across machines. Separate execution, waiting and retries; identify the critical path wherever work runs in parallel.

Next reproduce the problem under controlled load and insert a known delay at one location. If instrumentation identifies it correctly, compare hypotheses against actual slow cases. Change one mechanism whose effect can be verified: buffer draining, queue handling or view refresh. All three produce the same second in our example but require different interventions.

Acceptance of the fix should include sample size, end-to-end distribution, errors and lost events, offered load and resource impact. Retain the configuration and a case that was previously slow. Only that comparison supports a claim that user waiting was reduced, rather than merely the duration of whichever component was easiest to measure.

Sources

• OpenTelemetry: Traces, spans, context propagation and relationships between asynchronous operations.
• W3C: Trace Context, Recommendation dated 23 November 2021; HTTP context format and privacy constraints.
• Google SRE: Monitoring Distributed Systems, observing latency and the limits of averages.
• OpenTelemetry: Sampling, head and tail decisions and sampling costs.

Sources checked on 21 October 2026. The article explains principles and does not claim automatic instrumentation for a particular SDK. All durations, distributions and optimisation results come from a deterministic synthetic model.