Grafana Tempo’s datasource configuration for trace search is less about pointing to a specific trace and more about how Tempo itself indexes and retrieves trace data.
Let’s see it in action with a typical setup. Imagine you have a service instrumented with OpenTelemetry, sending traces to Tempo. In Grafana, you’d first add Tempo as a datasource.
Here’s a snippet of what that might look like in Grafana’s UI or its configuration file:
apiVersion: 1
datasources:
- name: Tempo
type: tempo
access: proxy
url: http://tempo-distributor:3200
basicAuth: false
isDefault: false
jsonData:
tracesToMetrics:
enabled: true
tags: true
search:
enabled: true
version: 1
The key here is the jsonData block, specifically search.enabled: true. This tells Grafana to enable the trace search functionality. When enabled, Tempo uses its indexing capabilities to allow you to query for traces based on attributes (like service name, operation name, or custom tags).
The problem Tempo solves is making distributed tracing searchable. Without it, you’d have to know the trace ID to retrieve a specific trace, which is impractical in a microservices environment where you’re debugging emergent issues. Tempo acts as a backend for Grafana’s explore view, allowing you to filter and find traces that match certain criteria.
Internally, Tempo achieves this by leveraging a combination of its components. When traces are sent to the Tempo distributor, they are then forwarded to the ingester. The ingester writes the trace data to object storage (like S3, GCS, or MinIO) and also generates index data. This index data is what Tempo’s query frontend uses to locate traces. The query frontend, in turn, communicates with the Tempo backend (which might be a single binary or multiple components depending on your deployment) to retrieve the actual trace spans from object storage.
The exact levers you control are primarily within the Tempo configuration itself, which influences how the index is built and queried. For instance, you can configure the tempo.ingester.trace_id_ratio to control how many trace IDs are indexed (a higher ratio means more trace IDs are indexed, leading to better searchability but more index storage). You also configure the tempo.query_frontend.default_max_api_response_items to limit the number of traces returned in a single search query.
Crucially, the search.enabled flag in the Grafana datasource configuration is a toggle for the frontend experience. It tells Grafana to ask Tempo for search capabilities. The actual indexing and retrieval logic is handled by Tempo’s backend components, and the effectiveness of search depends on how Tempo is configured to build its indexes. Without proper indexing configuration in Tempo itself (e.g., not setting up a suitable index store like Loki or Elasticsearch if you’re not using the default Boltdb), enabling search.enabled in Grafana might not yield useful results.
The next step in mastering Tempo is understanding how to optimize its indexing strategy for your specific workload and retention policies.