The Tempo Block Builder’s "Separate Worker Mode" is surprisingly not about distributing the work of building blocks, but about isolating the resource consumption of that work.
Let’s see it in action. Imagine you have a Tempo cluster with a dedicated Block Builder, and you’re ingesting a lot of traces. Without separate worker mode, the builder might consume CPU and memory that also impacts the query frontend, slowing down trace lookups.
Here’s a typical configuration snippet for Tempo:
# tempo/config.yaml
distributor:
receivers:
- "otlp"
ingester:
trace:
backend: "block"
# This is where the magic happens for the block builder
blocks:
# Default settings, might be shared
backend: "local"
local:
path: "/var/tempo/blocks"
# --- Separate Worker Mode Configuration ---
# This section is key when you want isolation.
# It's not a direct "worker_mode: true" but rather how you configure
# the components that *would* be running together.
# If you have a single Tempo binary and want to isolate the block builder's
# resource usage, you'd typically run it as a separate process or even a
# separate deployment. The configuration below assumes a single binary
# where you can logically separate concerns.
# In a true separate deployment scenario, you'd have a Tempo instance
# configured *only* with the necessary components for block building.
# For a single binary, the isolation comes from how you tune the JVM/Go runtime
# or container resource limits.
# Let's illustrate the *intent* of separation via tuning.
# These are not direct 'separate worker mode' flags, but how you'd
# configure a Tempo instance *dedicated* to block building if it were separate.
# For a dedicated block builder instance (or logical separation within one):
# These settings would be tuned for heavy write/processing load,
# potentially with less focus on low-latency reads.
# Example tuning for a dedicated block builder instance:
# (These are Go runtime tuning flags, often set via environment variables)
# GOMAXPROCS=4 # Limit CPU cores for this process
# TEMPO_INGESTER_TRACE_MAX_QUEUE_SIZE=10000 # Allow a larger buffer for incoming traces
# TEMPO_INGESTER_TRACE_BATCH_SIZE=1000 # Process traces in larger batches
# TEMPO_BLOCK_BUILDER_MAX_CONCURRENT_FLUSHES=8 # More flushes to disk concurrently
# If running as separate processes, you'd configure them like this:
# Process 1: Tempo Query Frontend (and Distributor/Ingester)
# tempo:
# ... standard config ...
#
# Process 2: Tempo Block Builder (Dedicated)
# tempo:
# ingester:
# trace:
# backend: "block"
# blocks:
# backend: "local"
# local:
# path: "/var/tempo/blocks"
# # Crucially, you'd limit resources here via container limits or OS tuning.
# # For example, in Kubernetes:
# # resources:
# # limits:
# # cpu: "2"
# # memory: "4Gi"
# # requests:
# # cpu: "1"
# # memory: "2Gi"
# The key is that the *work* of building blocks (compacting, flushing, etc.)
# is performed by a process or set of processes that don't have to compete
# with query processing resources.
The core problem Tempo solves is efficient storage and retrieval of trace data. Traces are often bursty and can generate massive amounts of data. Storing them raw is inefficient. Tempo’s block builder’s job is to take these incoming traces, group them into "blocks" (which are essentially optimized, compressed on-disk files), and manage their lifecycle (compaction, deletion).
The "Separate Worker Mode" isn’t a specific flag you toggle. Instead, it’s an operational pattern achieved by running the Tempo components responsible for block building (primarily the ingester’s trace backend and the blocks configuration) as a distinct set of processes or even a separate Tempo deployment. This allows you to dedicate specific CPU, memory, and I/O resources to the computationally intensive task of building and flushing blocks, preventing it from starving the components responsible for serving queries.
Think of it like a busy restaurant. Without separation, the chefs (block builders) are also trying to serve tables (query frontends). If a big catering order (high trace ingestion) comes in, the chefs get overwhelmed, and table service suffers. By having dedicated kitchen staff (separate block builder processes), the chefs can focus on preparing food, and the waitstaff (query frontend) can focus on serving.
The mental model is about resource partitioning. You have the Tempo binary, which can run multiple components: distributor, ingester, querier, and the block builder logic (which is part of the ingester’s trace backend and the blocks configuration). When these run on the same process or same set of Kubernetes pods, they share the same CPU, memory, and network. If the block builder is churning through a large backlog of traces, it might consume 100% of a CPU core, making the querier slow to respond.
By "separating" the block builder, you’re essentially saying: "This set of Tempo processes will only do block building." You’d configure these processes with specific resource requests and limits (e.g., in Kubernetes) and potentially tune their internal settings (like GOMAXPROCS or ingester queue sizes) for maximum throughput on block construction, not for low-latency query response. The other Tempo processes (distributor, ingester for query-facing parts, querier) would then run on their own set of resources, free from the block builder’s heavy lifting.
The one thing most people don’t realize is that the "block builder" isn’t a single, distinct binary you can just download and run. It’s a role that a Tempo process can play, and the separation is an architectural choice you make by deploying and configuring one or more Tempo instances specifically for that role. You might have a cluster of Tempo instances acting as distributors and ingesters (handling incoming traces and preparing them for the block builder), and a separate cluster of Tempo instances configured only with the blocks backend settings, acting as dedicated block builders.
The next logical step after achieving this isolation is optimizing the network bandwidth and disk I/O for the dedicated block builder nodes.