Grafana Tempo’s external search index, when using Apache Parquet, fundamentally changes how you query trace data by moving from an in-memory index to a disk-based, columnar format.
Let’s see this in action. Imagine you have Tempo configured to use Parquet for its index, and you’re querying for traces from a specific service, user-service, within the last hour.
curl -G "http://localhost:3100/api/search" \
--data-urlencode 'query={service="user-service"}' \
--data-urlencode 'start=1678886400000' \
--data-urlencode 'end=1678890000000'
When this request hits Tempo, it doesn’t scan an in-memory map of trace IDs. Instead, it interacts with your object storage (like S3 or GCS) where your Parquet files reside. Tempo’s query frontend will consult its local index of Parquet files, understanding which files contain metadata relevant to service="user-service" and the specified time range. It then dispatches requests to the Tempo ingesters (or directly to object storage, depending on configuration) to read only the necessary columns from those Parquet files. This selective reading of data is the core of Parquet’s efficiency.
The problem Tempo solves is scaling trace storage and query performance. As trace volumes grow, keeping an index entirely in memory becomes prohibitively expensive and eventually impossible. Tempo needs a way to persist this index and query it efficiently, even when it’s larger than RAM. Apache Parquet, a columnar storage file format, provides this solution. Instead of storing data row by row, Parquet stores data column by column. This means that when you query for a specific set of attributes (like service and span_name), Tempo only needs to read the service and span_name columns from the Parquet files, dramatically reducing I/O compared to row-based formats.
Internally, Tempo writes its index data (trace IDs, service names, operation names, and timestamps) to Parquet files. These files are organized and stored in your chosen object storage. When a query comes in, Tempo uses a bloom filter or other indexing structures within the Parquet files themselves to quickly locate relevant data blocks without reading entire files. The query frontend coordinates this, identifying which Parquet files are likely to contain the traces you’re looking for based on metadata and time ranges, and then directs the actual data retrieval to the appropriate backend components or directly to object storage.
The exact levers you control are primarily around how Tempo writes and manages these Parquet files. This includes:
ingester.max-block-size: This setting in Tempo’s configuration dictates the maximum number of index entries (traces, spans) that will be batched together before being written to a Parquet file. A smaller block size can lead to more, smaller files, which might increase object storage operations overhead but can also improve query latency if queries hit only a few blocks. A larger block size means fewer, larger files, potentially reducing object storage operations but increasing the amount of data scanned per query if a query spans multiple blocks.index.parquet.storage-path: This defines the base path in your object storage where Tempo will write its Parquet index files. You’ll typically configure this with your S3 bucket, GCS path, or other object storage endpoint.index.parquet.flush-interval: How often Tempo attempts to flush its current in-memory index buffer to Parquet files. A shorter interval means fresher index data available for querying but more frequent writes. A longer interval reduces write frequency but means older index data might not be immediately queryable.
The most surprising part of Parquet’s efficiency for trace indexing is how it handles metadata. Each Parquet file contains metadata about the data within it, including statistics for each column (min/max values, null counts). Tempo leverages these statistics to prune entire files from consideration before even attempting to read data. If a Parquet file’s metadata indicates that its timestamp column has a maximum value earlier than your query’s start time, Tempo will skip reading that file entirely. This predicate pushdown at the file level is crucial for performance on large datasets.
The next logical step after understanding Parquet indexing is exploring how Tempo handles data retention and compaction for these Parquet files.