You can find a specific trace in Grafana Tempo by its ID, but it’s not always as straightforward as you’d think.
Let’s say you’ve got a trace ID, f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6, and you want to see just that trace in Tempo. You’d typically go to the Explore view in Grafana, select your Tempo data source, and in the query bar, you’d type something like:
{traceID="f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6"}
And hit "Run query." Tempo will then query its backends (like Loki for logs, or its own internal index if you’re using that) to find all spans associated with that trace ID and display them.
Here’s a Tempo instance running, showing a query for a specific trace ID:
{
"status": "success",
"data": {
"resultType": "logs",
"result": [
{
"value": [
1678886400000000000,
"level=\"info\" caller=\"server/server.go:123\" msg=\"handling request\" traceID=\"f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6\" method=\"GET\" path=\"/users\"\n"
],
"stream": {
"filename": "/var/log/app.log",
"job": "my-app"
}
},
{
"value": [
1678886401500000000,
"level=\"debug\" caller=\"user_service/service.go:45\" msg=\"user fetched\" traceID=\"f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6\" userID=\"123\"\n"
],
"stream": {
"filename": "/var/log/app.log",
"job": "my-app"
}
},
{
"value": [
1678886402800000000,
"level=\"info\" caller=\"api_gateway/gateway.go:88\" msg=\"response sent\" traceID=\"f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6\" status=\"200\"\n"
],
"stream": {
"filename": "/var/log/app.log",
"job": "my-app"
}
}
]
}
}
This JSON represents logs that Tempo has retrieved. Each entry includes a timestamp, the log message (which contains the traceID), and metadata about the log source (stream). Tempo correlates these logs and spans using the traceID.
The core problem Tempo solves is making sense of distributed systems. When a request travels across multiple services, a single trace ID stitches together all the individual pieces of work (spans) that contributed to that request. Without it, debugging a cascading failure or performance bottleneck would be like trying to find a single grain of sand on a beach. Tempo, by ingesting and indexing these spans, provides a unified view of that entire journey.
Internally, Tempo relies on a few key components. The collector receives spans from your applications (often via the OpenTelemetry protocol). These spans are then sent to ingesters, which write them to object storage (like S3, GCS, or MinIO). Crucially, Tempo also has an index component. This index is what allows you to quickly search for traces by their IDs or other attributes. The index can be built directly by Tempo, or it can leverage an external system like Loki, where logs are indexed, and Tempo queries Loki for log lines that contain span IDs. The query frontend and query services then retrieve and assemble the spans from object storage based on index lookups.
The levers you control are primarily in how you configure your Tempo instance and how your applications emit traces. For Tempo itself, you’ll configure its storage backend, its indexing strategy (e.g., using its internal index or integrating with Loki), and its network ports. For your applications, you’ll use OpenTelemetry SDKs to ensure they are correctly configured to export traces, including the traceID, to your Tempo collector. This often involves setting environment variables like OTEL_EXPORTER_OTLP_ENDPOINT.
When you query Tempo for a specific traceID, the process involves Tempo’s query service first consulting its index. If the index is configured to use Loki, Tempo will issue a Loki query like {traceID="f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6"}. Loki, in turn, will scan its index for log streams matching that label. Once Loki returns the relevant log lines (which Tempo treats as spans in this context), Tempo will then fetch the full span data from its object storage, if necessary, to reconstruct the complete trace visualization. The speed of this lookup is entirely dependent on how effectively the traceID is indexed and how quickly the underlying storage (object storage for spans, and Loki’s index for logs) can be queried.
The most surprising thing to many is how Tempo’s internal index and Loki’s index are fundamentally different in their approach to storing trace data, even though they can be used together. Tempo’s internal index is designed for trace-specific lookups, storing derived data like trace IDs and service names. Loki, on the other hand, is a log aggregation system that indexes log lines and their associated labels. When Tempo uses Loki for indexing, it’s essentially treating log lines that contain trace IDs as if they were spans, relying on Loki’s efficient label-based querying to find them. This means that for trace ID lookups via Loki, the traceID must be present as a label (or part of the log message that Loki indexes as a label) in the logs being sent to Loki.
The next step after mastering trace ID lookups is exploring how to correlate metrics and logs with traces using Grafana’s unified observability features.