The most surprising thing about Tempo’s Blocks Processor is that its primary job isn’t really about aggregation at all; it’s about making sure your traces don’t get lost in the shuffle of writes and reads.
Let’s watch it work. Imagine a tiny, fictional application writing spans.
// Span 1: Start of request
{
"traceID": "abc123xyz",
"spanID": "111",
"parentSpanID": "",
"operationName": "GET /users",
"startTime": 1678886400000000000,
"endTime": 1678886400100000000,
"tags": {"http.method": "GET", "http.status_code": 200}
}
// Span 2: Database query for user data
{
"traceID": "abc123xyz",
"spanID": "222",
"parentSpanID": "111",
"operationName": "db.query",
"startTime": 1678886400050000000,
"endTime": 1678886400070000000,
"tags": {"db.system": "postgresql"}
}
// Span 3: Another request
{
"traceID": "def456uvw",
"spanID": "333",
"parentSpanID": "",
"operationName": "POST /orders",
"startTime": 1678886401000000000,
"endTime": 1678886401200000000,
"tags": {"http.method": "POST", "http.status_code": 201}
}
When these spans arrive at Tempo, they don’t immediately get stored as individual, queryable units. Instead, they are batched and written to object storage (like S3, GCS, or MinIO) in a format called "blocks." The Blocks Processor, running within the Tempo distributor or querier, is the component responsible for managing these blocks. It doesn’t magically combine unrelated spans; rather, it ensures that all spans belonging to a single trace are grouped together efficiently for retrieval.
The core problem Tempo solves is storing and retrieving massive volumes of trace data without needing a complex, relational database for every single span. It achieves this by writing traces as immutable blocks to object storage. The Blocks Processor’s role is to take incoming spans, figure out which trace they belong to, and ensure they are written into the correct block. When a query comes in for a specific traceID, the Blocks Processor uses its internal index (which is also persisted) to locate the relevant blocks and then retrieves them for the querier. It’s essentially a highly optimized write-and-index mechanism for trace data.
Here’s how you configure the Blocks Processor. In your Tempo configuration (tempo.yaml), you’ll find the blocks section.
# tempo.yaml
blocks:
backend:
s3:
# S3 bucket configuration
bucket: "my-tempo-traces"
endpoint: "s3.amazonaws.com"
region: "us-east-1"
access_key_id: "YOUR_ACCESS_KEY_ID"
secret_access_key: "YOUR_SECRET_ACCESS_KEY"
# Controls how often blocks are flushed to storage.
# Shorter intervals mean less data loss on crash but more writes.
# Longer intervals mean fewer writes but more potential data loss.
flush_interval: "10m"
# Controls the maximum size of a block before it's flushed.
# Larger blocks can be more efficient for storage but slower to retrieve.
max_block_bytes: 536870912 # 512 MiB
# Controls how long spans are held in memory before being flushed.
# This is a safeguard against extremely long-running traces.
max_block_duration: "1h"
# Controls how many blocks can be open for writing concurrently.
# Affects throughput under heavy load.
max_concurrent_writes: 16
The flush_interval dictates how frequently the processor attempts to write out a completed block of spans. A value like 10m means that after 10 minutes, or when max_block_bytes or max_block_duration is reached, the current block will be finalized and written to object storage. max_block_bytes sets the upper limit for a single block’s size in bytes (here, 512 MiB). max_block_duration is a timeout for how long spans for a single trace can be buffered before being flushed, preventing unbounded memory usage for very long-lived traces. max_concurrent_writes limits how many blocks can be actively written to object storage simultaneously, preventing overwhelming the storage backend.
The Blocks Processor doesn’t perform complex trace assembly or aggregation in the sense of summing up metrics. Its "aggregation" is purely structural: it groups spans belonging to the same trace into a single block file for efficient storage and retrieval. When you query for a traceID, Tempo’s querier asks the Blocks Processor to find and return the block(s) containing that trace. The processor consults its internal index, which maps traceIDs to the specific block files they reside in, and then fetches those files from object storage.
What most people miss is that Tempo’s internal indexing for block location isn’t a single, monolithic index. Instead, each block file itself contains metadata that points to other blocks or provides sufficient information for the querier to locate all parts of a trace, especially when a trace spans multiple blocks due to size or time limits. This distributed indexing within the blocks themselves is key to Tempo’s scalability.
The next thing you’ll likely encounter is how Tempo handles compaction of these blocks to reduce storage costs and improve read performance.