Understanding a Span in Distributed Tracing
A span represents one unit of work within a distributed trace, such as handling an HTTP request, executing a database query, or calling a downstream service. While distributed tracing as a whole is the system for following a request across services, the span is the atomic building block that tracing systems like Jaeger and Tempo actually store, index, and visualize. Understanding the structure of a span, rather than just the concept of tracing, is essential for reading a trace waterfall diagram, writing manual instrumentation, or debugging why a trace looks incomplete.
The Structure of a Span
Every span, as defined by the OpenTelemetry specification, carries a consistent set of fields:
- Trace ID: a globally unique identifier shared by every span that belongs to the same trace, letting a tracing backend group them together.
- Span ID: a unique identifier for this specific span, distinct from every other span even within the same trace.
- Parent Span ID: the span ID of the operation that caused this span to start, which is what allows a tracing UI to reconstruct the parent-child hierarchy. A span with no parent ID is the root span of the trace.
- Name: a human-readable operation name, such as GET /checkout or SELECT orders.
- Start time and duration: precise timestamps that let a tracing UI draw the span as a bar on a timeline, showing exactly how long the operation took and when it started relative to its parent.
- Attributes (tags): key-value metadata describing the operation, such as http.status_code, db.system, or a custom business attribute like order.id, following OpenTelemetry’s semantic conventions where possible.
- Events: timestamped annotations within the span’s lifetime, such as marking the moment a retry occurred or an exception was caught.
- Status: whether the span completed successfully, with an error, or is unset, which tracing UIs use to color spans red when something failed.
How Spans Form a Trace
When a request enters a system, the first service creates a root span with a new trace ID and no parent. As that service calls another service, it creates a child span, propagates the trace ID and its own span ID (as the new parent ID) via headers like traceparent, and the downstream service uses that context to create its own child span. This chain continues across every service the request touches. A tracing backend like Jaeger collects all of these spans, groups them by trace ID, and uses the parent-child relationships to render a waterfall diagram showing which operations ran sequentially, which ran in parallel, and where time was actually spent.
A Concrete Example
A checkout request produces a root span named POST /checkout lasting 850 milliseconds. Inside it, a child span for validate-cart takes 40 milliseconds, a child span for charge-payment takes 700 milliseconds and has an attribute payment.gateway set to “stripe”, and a child span for reserve-inventory takes 90 milliseconds and runs in parallel with the payment call. Looking at the waterfall, an engineer immediately sees that charge-payment dominates the total latency, and can drill into that span’s attributes and events to see it made a retry after an initial timeout, which is recorded as an event with a timestamp partway through the span’s duration.
Best Practices for Working with Spans
- Name spans consistently and predictably, typically after the operation being performed, so traces are readable without needing tribal knowledge of the codebase.
- Attach attributes using OpenTelemetry semantic conventions rather than inventing ad hoc keys, so spans are consistent and comparable across services and teams.
- Avoid attaching high-cardinality or sensitive data as span attributes, since trace storage backends index and retain these values.
- Use span events to capture discrete occurrences within a long operation, such as a retry or a cache miss, rather than creating an excessive number of tiny child spans.
- Ensure trace context (trace ID and parent span ID) is propagated correctly across every service boundary and async boundary, since a broken propagation link produces an orphaned trace instead of a single connected one.
Frequently Asked Questions
What is Span (in Distributed Tracing)?
A span is the basic unit of work in distributed tracing, representing a single operation such as an HTTP call or a database query, with its own ID, timing, and metadata. A trace is made up of one or more spans connected by parent-child relationships that reconstruct a request's full path through a system.
How does Span (in Distributed Tracing) work?
Span (in Distributed Tracing) works by combining the components described in the sections above. The main page walks through the architecture, the typical use cases, and the trade-offs to weigh before adopting it.
Why does Span (in Distributed Tracing) matter?
Teams adopt Span (in Distributed Tracing) to ship faster, run more reliably, and reduce the cognitive load on engineers. The benefits, limits, and adjacent tools are covered in the body above.
When should you use Span (in Distributed Tracing)?
Use Span (in Distributed Tracing) when the problems it solves match what your team is hitting today. The page above outlines the signals that mean you should adopt it now, and the cases where a simpler approach is fine.
