Understanding a Flame Graph
A flame graph is a way of visualizing profiling data, originally popularized by performance engineer Brendan Gregg, that turns thousands of raw stack trace samples into a single readable image. Rather than presenting a table of function names and sample counts, a flame graph arranges functions as stacked, horizontal bars: the y-axis represents stack depth, with the function at the bottom being the entry point (such as main or a request handler) and each function it calls stacked above it, while the x-axis width of each bar represents the proportion of total samples in which that function was on the call stack, not the passage of time from left to right.
How to Read a Flame Graph
- Width equals frequency, not time order. A wide bar means that function consumed a large share of the profiled samples, whether that’s CPU time or memory allocations; the horizontal position has no chronological meaning, unlike a trace waterfall diagram.
- Height equals call stack depth. Reading from bottom to top shows the chain of function calls, for example a handler calling a serializer which calls a compression function.
- Color is typically arbitrary or used for grouping, such as distinguishing application code from library code, and generally does not encode a specific metric on its own.
- An inverted variant, sometimes called an icicle graph, flips the orientation so the root is at the top and children hang below it, which some tools prefer for readability.
A Concrete Example
Suppose a Go service’s CPU usage is unexpectedly high. A flame graph generated from a continuous profiling tool like Pyroscope or Parca shows a wide bar near the bottom for the HTTP handler, as expected since it wraps everything, but a surprisingly wide bar higher up for a function called normalizeAddress, taking up 35 percent of total sampled width. Drilling in, the engineer sees that normalizeAddress calls a regular expression compilation function on every single invocation instead of caching the compiled pattern, meaning every request pays the cost of recompiling the same regex. The flame graph made this obvious in seconds, whereas reading through the code alone might never have surfaced this as the actual bottleneck, since the code “looks” fine at a glance.
Where Flame Graphs Come From
Flame graphs are generated from stack trace samples collected by a profiler, whether a one-off tool like Go’s pprof, Python’s py-spy, or a continuous profiling system that samples production processes on an ongoing basis. The raw samples are aggregated into a tree structure counting how often each unique call stack path appeared, and that tree is what gets rendered as the flame graph. Because generating a flame graph only requires stack samples and their frequency, the same visualization works for CPU profiles, memory allocation profiles, and even off-CPU time such as time spent blocked on I/O or lock contention.
Trade-offs and Limitations
- Flame graphs show aggregate proportions over a sampling window; they don’t show when during that window the expensive calls happened, which is what a trace timeline is better suited for.
- Very deep call stacks or highly recursive code can produce flame graphs that are visually cluttered and hard to read without interactive zooming.
- Reading a flame graph accurately requires understanding that width reflects sample frequency, not wall-clock duration, which is a common point of confusion for engineers new to profiling.
Best Practices
- Use a diff flame graph, comparing two time windows or two versions of code, to isolate exactly what changed rather than eyeballing a single graph in isolation.
- Combine flame graphs with distributed tracing: a slow trace span points to which service and operation is slow, and a flame graph for that service during the same window shows which function inside it is responsible.
- Generate flame graphs from continuous profiling data rather than only during manual investigation, so historical incidents can be profiled retroactively.
- Pay attention to the widest bars first since they represent the highest-leverage optimization targets, rather than getting lost in narrow, low-impact branches of the graph.
Frequently Asked Questions
What is Flame Graph?
A flame graph visualizes stack trace samples as horizontal bars whose width shows how often each function appeared in profiled call stacks. It lets engineers instantly spot which functions consume the most CPU or memory without reading raw profiling data line by line.
How does Flame Graph work?
Flame Graph works by combining the components described in the sections above. The main page walks through the architecture, the typical use cases, and the trade-offs to weigh before adopting it.
Why does Flame Graph matter?
Teams adopt Flame Graph to ship faster, run more reliably, and reduce the cognitive load on engineers. The benefits, limits, and adjacent tools are covered in the body above.
When should you use Flame Graph?
Use Flame Graph when the problems it solves match what your team is hitting today. The page above outlines the signals that mean you should adopt it now, and the cases where a simpler approach is fine.
