Understanding Trace Sampling
Trace sampling is the practice of deciding which requests get fully recorded as traces and which are discarded, applied because capturing and storing a complete trace for every single request in a high-traffic system is often prohibitively expensive. A service handling 50,000 requests per second, each producing a trace with a dozen spans, would generate an enormous volume of tracing data if every request were retained; sampling reduces this to a manageable and representative subset while still preserving the ability to debug problems.
Head-Based Sampling
Head-based sampling makes the sampling decision at the very start of a request, typically at the entry point, before any of the request’s downstream behavior is known. The most common form is probabilistic sampling, for example keeping 1 percent or 10 percent of all traces at random. This decision, along with the trace ID, is propagated through the W3C Trace Context headers to every downstream service, so all services agree on whether this particular trace is being sampled. Head-based sampling is simple to implement and has low overhead since unsampled requests never generate spans in the first place, but it has a significant blind spot: because the decision is made before anything goes wrong, a purely random 1 percent sample will capture very few of the rare, interesting traces, like the ones that end in an error or have unusually high latency.
Tail-Based Sampling
Tail-based sampling instead waits until an entire trace has completed, buffering all its spans temporarily, then decides whether to keep it based on characteristics of the complete trace, such as whether it contains an error, whether its total duration exceeds a latency threshold, or other custom criteria. This means engineers can specify a policy like “always keep traces with errors, always keep traces slower than 2 seconds, and sample 5 percent of everything else,” which preserves the most valuable evidence while still controlling volume. The OpenTelemetry Collector supports tail-based sampling through a dedicated processor, though it requires buffering spans in memory until a trace is complete, which adds latency and resource overhead compared to head-based sampling.
A Concrete Example
A payments service samples traces at a flat 2 percent head-based rate to control cost. During an incident where 0.5 percent of payment requests are failing, the flat sampling means only 1 in 100 of those failing requests actually gets captured as a trace, roughly 1 in 20,000 total requests, making it very difficult to gather enough failed traces to diagnose the pattern. Switching to a tail-based sampling policy that always retains error traces and slow traces, while sampling only a small fraction of fast, successful ones, ensures that every failure is captured for analysis regardless of how rare it is, while overall trace volume and storage cost remain controlled.
Trade-offs
- Head-based sampling is cheaper and simpler but risks missing rare but important traces unless the sampling rate is high enough, which defeats the cost-saving purpose.
- Tail-based sampling captures the traces that matter most but requires buffering full traces in a collector, adding memory overhead and latency to the export pipeline, and requires all spans of a trace to route through a collector capable of making the tail decision.
- Both approaches can introduce sampling bias if not configured carefully, undercounting genuine but non-extreme variations in behavior.
Best Practices
- Use tail-based sampling, via the OpenTelemetry Collector, for systems where capturing every error and every slow request matters most, and accept the added infrastructure complexity.
- Combine a low baseline probabilistic sampling rate with guaranteed retention of error and high-latency traces to balance cost against debugging completeness.
- Ensure the sampling decision and trace context propagate consistently across all services in a request chain so a trace isn’t partially recorded by some services and dropped by others.
- Revisit sampling rates as traffic grows, since a sampling rate tuned for one traffic level may become too sparse or too expensive as the system scales.
Frequently Asked Questions
What is Trace Sampling?
Trace sampling records only a subset of distributed traces a system generates, rather than every request, to control the storage and performance cost of tracing at scale. Decisions can be made at request start (head-based) or after seeing the full trace (tail-based).
How does Trace Sampling work?
Trace Sampling works by combining the components described in the sections above. The main page walks through the architecture, the typical use cases, and the trade-offs to weigh before adopting it.
Why does Trace Sampling matter?
Teams adopt Trace Sampling to ship faster, run more reliably, and reduce the cognitive load on engineers. The benefits, limits, and adjacent tools are covered in the body above.
When should you use Trace Sampling?
Use Trace Sampling when the problems it solves match what your team is hitting today. The page above outlines the signals that mean you should adopt it now, and the cases where a simpler approach is fine.
