Tempo’s tag value search is surprisingly powerful, but it doesn’t work like you’d expect based on other tracing backends or even Grafana’s own dashboarding.
Let’s say you’re looking for traces where a specific service, my-service, made a call to another service, another-service, and you want to filter by the HTTP method. You might try something like this in Tempo’s query editor:
{service="my-service", http.method="POST"}
But that’s not how Tempo rolls. Tempo’s primary indexing is on service name and operation name. Tag values are not indexed for direct querying in the same way. When you query for http.method="POST", Tempo is performing a full scan of all spans that match service="my-service" and then filtering those results for spans that also have the http.method attribute set to POST. This is why it feels slow or sometimes doesn’t return results when you expect it to.
The mental model you need for Tempo search is this:
- Service & Operation are King: Tempo’s main indexing is built around
serviceandoperationnames. These are highly efficient to query. - Attributes are Secondary: Arbitrary key-value attributes (like
http.method,db.system,user.id) are stored with spans but are not part of the primary index. When you filter by an attribute, Tempo first finds all traces matching the service/operation, and then iterates through the spans within those traces to find matches for your attribute filter. - TraceQL is the Way: Tempo’s query language, TraceQL, is designed to leverage this structure.
Let’s see it in action. Imagine you have traces from frontend service calling backend service.
Here’s a basic query for all traces from frontend:
{service="frontend"}
This will be fast because service="frontend" is directly indexed.
Now, let’s say you want to find traces where frontend called backend, and you want to filter by a specific trace ID.
{service="frontend", traceID="abc123def456"}
This is still efficient as traceID is also a primary lookup.
The challenge comes when you want to filter by span attributes. Suppose you want traces from my-service that performed an HTTP POST request.
{service="my-service"} | http.method="POST"
This is the correct TraceQL syntax. The first part {service="my-service"} hits the primary index. The | http.method="POST" part then filters the spans within the traces returned by the first part. If you have many traces from my-service and many of them have http.method attributes, this second stage of filtering can be computationally expensive.
To make this more performant, you need to leverage Tempo’s ability to index specific attributes. This is done via the tempo-distributor configuration. You can define which attributes should be indexed for faster lookups.
For example, in your tempo-distributor.yaml:
common:
path:
# ... other path configurations
service_name: tempo-distributor
distributor:
receivers:
# ... other receivers
ingestion:
max_path_get_retries: 3
max_trace_id_retries: 3
max_span_id_retries: 3
# Index specific attributes for faster searching
span_attributes:
- service.name # This is usually indexed by default
- http.method
- db.system
- http.status_code
When you add http.method to span_attributes in the distributor configuration and restart Tempo, new spans ingested will have their http.method values indexed. This means queries like {service="my-service"} | http.method="POST" will become significantly faster because Tempo can now directly look up spans with http.method="POST" within the my-service traces, rather than scanning all spans.
The most surprising thing about Tempo’s attribute indexing is that it’s not enabled by default for arbitrary attributes. You have to explicitly tell Tempo which attributes are important to you for searching. This is a deliberate design choice to keep ingestion performance high by default, and allows users to tailor indexing to their specific observability needs. If an attribute isn’t listed in span_attributes, Tempo will perform a full scan of matching spans for that attribute filter.
The next concept you’ll likely explore is how to combine attribute filters with other TraceQL operators like -> for parent-child relationships or ~ for regular expression matching on span names.