Grafana Tempo’s ingester is designed to buffer incoming traces in memory before asynchronously flushing them to backend object storage, like S3 or GCS.
Let’s see this in action. Imagine we’re sending traces from a service to Tempo.
# Example trace data (simplified OpenTelemetry format)
{
"resourceSpans": [
{
"resource": {
"attributes": [
{
"key": "service.name",
"value": {
"stringValue": "my-backend-service"
}
}
]
},
"scopeSpans": [
{
"spans": [
{
"traceId": "a1b2c3d4e5f67890a1b2c3d4e5f67890",
"spanId": "0102030405060708",
"name": "process_request",
"startTimeUnixNano": 1678886400000000000,
"endTimeUnixNano": 1678886400100000000,
"kind": "SPAN_KIND_SERVER",
"attributes": [
{
"key": "http.method",
"value": {
"stringValue": "GET"
}
}
]
}
]
}
]
}
]
}
When this trace data arrives at the Tempo ingester, it doesn’t immediately write it to object storage. Instead, it holds onto it.
Internally, the ingester uses a process called "compaction." It groups traces that share the same trace ID and are within a certain time window. This is crucial because object storage operations, especially writes, can be expensive. By grouping traces, Tempo can write larger, more efficient chunks to storage.
The primary levers you control are related to this buffering and compaction process. The tempo.ingester.max-block-size setting determines the maximum size of a block of traces that the ingester will try to write to storage at once. A larger block size can lead to fewer, larger writes, potentially improving throughput. Conversely, tempo.ingester.max-block-duration defines the maximum time a block will be held in memory before being flushed, even if it hasn’t reached the max-block-size. This ensures that traces don’t stay in memory indefinitely.
The ingester also has tempo.ingester.max-in-memory-traces which limits the total number of traces held in memory across all ingesters. This is a hard safety net to prevent memory exhaustion. If this limit is reached, new traces will be rejected until space is freed up.
The real magic happens when a block reaches its maximum size or duration. Tempo then serializes this block into a format (like Apache Parquet) and uploads it as a single object to your configured backend storage. This is why Tempo is so cost-effective for long-term trace retention: it leverages the economics of object storage by writing large, immutable files.
What most people don’t realize is how the tempo.ingester.trace-idle-period configuration directly impacts the end of a trace. If a trace has no new spans for longer than this duration, Tempo considers the trace "closed" and will eventually include its final segment in the next flush. This is essential for correctly reconstructing entire traces, as a trace might span multiple requests or asynchronous operations.
When you configure your Tempo ingester, you’re essentially tuning the trade-off between latency (how quickly traces appear in storage) and throughput/efficiency (how many traces you can process with available resources).
The next hurdle you’ll likely encounter is understanding how Tempo’s query frontend interacts with this stored data.