Tempo’s tempo-cli can directly inspect and debug the internal block and trace data it stores, giving you a low-level view of what’s actually on disk.
Let’s see what’s inside a Tempo block file. Assume you have a block file named 01GZ4C7Q044Y5X9N7Q1B3X7T2F.brt in your current directory.
tempo-cli block inspect 01GZ4C7Q044Y5X9N7Q1B3X7T2F.brt
This command will output a JSON representation of the block’s metadata. You’ll see information like the block ID, creation time, and importantly, the list of indexed traces within that block. Each trace entry will include its ID, start and end times, and the set of associated Span IDs. This is useful for understanding how Tempo groups traces and what metadata it keeps for quick lookups.
Now, let’s say you want to see the actual trace data contained within that same block file.
tempo-cli block dump-traces 01GZ4C7Q044Y5X9N7Q1B3X7T2F.brt
This command will output the raw trace data in the Tempo Trace format. You’ll see a stream of trace objects, each containing its spans. This is invaluable for debugging issues where traces aren’t appearing correctly in the UI or if you suspect data corruption at the lowest level. You can pipe this output to jq for easier parsing:
tempo-cli block dump-traces 01GZ4C7Q044Y5X9N7Q1B3X7T2F.brt | jq '.'
When Tempo writes data, it serializes spans into a compressed format within a block. The inspect command reads the block’s index, which is a separate part of the file, allowing it to quickly list what’s inside without decompressing all the trace data. The dump-traces command, however, must decompress and deserialize the actual span data for each trace stored in the block.
What if you have a specific trace ID and want to find which block(s) it resides in and then inspect that block? You can use tempo-cli trace query. Let’s say your trace ID is a1b2c3d4e5f6a7b8.
tempo-cli trace query a1b2c3d4e5f6a7b8
This command will query Tempo’s index (if you have a Tempo instance running and configured to use an index, or if you’re pointing to a local object store containing index data) and return the block IDs that contain the specified trace. The output might look like this:
Blocks:
- 01GZ4C7Q044Y5X9N7Q1B3X7T2F
Once you have the block ID, you can then use tempo-cli block inspect or tempo-cli block dump-traces as shown previously, by first downloading the corresponding .brt file from your object storage.
If you’re dealing with an issue where specific spans or entire traces are missing, but you know they should be there, examining the block files directly with tempo-cli is your first step. It bypasses the Tempo query engine and shows you exactly what Tempo has persisted. This is crucial for differentiating between a query problem (e.g., incorrect time range, wrong service name) and a storage problem (e.g., data not written, data corrupted, block unreadable).
Consider a scenario where you’ve recently upgraded Tempo, and traces from before the upgrade are intermittently visible. You might suspect a change in how blocks are written or read. Using tempo-cli block inspect on older .brt files can reveal differences in indexing or metadata compared to newer blocks. If the older blocks are unreadable by tempo-cli with a newer version, it points to a potential backward compatibility issue in the block format itself.
Another common debugging path involves checking the integrity of the block files themselves. If your object storage reports errors or if Tempo itself is failing to read blocks, you can use tempo-cli block verify to check for corruption.
tempo-cli block verify 01GZ4C7Q044Y5X9N7Q1B3X7T2F.brt
This command performs checksums and structural checks on the block file. If it reports errors, the block file is likely corrupted and may need to be re-ingested or may be unrecoverable.
The tempo-cli commands operate directly on the .brt files, which are the fundamental unit of storage for Tempo’s block storage. These files are essentially compressed archives containing serialized trace data and an index for fast lookups. When Tempo queries for traces, it first consults its index (which can be in object storage, a database, or in-memory depending on configuration) to identify relevant blocks. Then, it fetches those blocks and uses their internal indexes to locate the specific traces or spans. The tempo-cli commands simulate this process at a granular level, allowing you to examine the data Tempo itself sees.
The trace IDs you see in the inspect output are derived from the spans within the block. Tempo uses a deterministic algorithm based on the span’s trace ID and its position within the block to assign an internal "indexed trace ID" for faster lookups. When you query for a trace ID, Tempo maps that to the relevant indexed trace IDs and then finds the corresponding blocks. The dump-traces command essentially reverses the serialization process that Tempo performs during ingestion, giving you the raw span data as it was originally sent or processed.
The next thing you’ll likely run into when debugging block-level issues is understanding how blocks are distributed across your object storage and how Tempo manages their lifecycle (e.g., compaction, deletion).