Tempo’s per-tenant overrides let you customize tracing ingestion and retention settings for individual tenants, but they’re not a free-for-all; they’re strictly bound by the global configuration.
Let’s see how this plays out with a concrete example. Imagine you have a central Tempo cluster serving multiple teams, and you want Team A to have a higher trace ingestion rate and longer retention than others.
// Global Tempo Config (tempo.yaml)
ingester:
max_traces_per_second: 10000
trace_idle_period: "1h"
max_block_duration: "1h"
block_storage_type: "s3"
s3:
bucket: "tempo-us-central1-prod-us-central1-bucket"
endpoint: "s3.us-central1.amazonaws.com"
compactor:
shared_store: "s3"
s3:
bucket: "tempo-us-central1-prod-us-central1-bucket"
endpoint: "s3.us-central1.amazonaws.com"
distributor:
receivers: ["otlp"]
querier:
query_frontend_address: "tempo-query-frontend:8080"
ui:
default_tenant_id: "default"
Now, you want to set up per-tenant overrides for "team-a". You’d typically do this via the Tempo API or a configuration management system that can dynamically update Tempo’s configuration or the configuration of its components like the distributor.
Here’s how you might configure overrides (though the exact mechanism depends on your deployment, often it’s via a dynamic configuration endpoint or a separate config file reloaded by the distributor):
// Per-Tenant Override Example (conceptually)
// This is NOT a direct tempo.yaml snippet, but represents the idea of per-tenant config.
// In practice, this might be managed via an API or a separate dynamic configuration file reloaded by the distributor.
{
"tenants": {
"team-a": {
"ingester": {
"max_traces_per_second": 20000, // Higher than global
"trace_idle_period": "2h" // Longer than global
},
"limits": {
"max_traces_per_second": 20000 // Must be <= global max_traces_per_second
}
}
}
}
The core problem Tempo solves is the efficient storage and retrieval of distributed traces. Traces can generate a massive amount of data, and Tempo is designed to handle this by breaking traces into blocks, compressing them, and storing them in object storage like S3. The distributor receives traces, the ingester writes them to storage, and the querier retrieves them. Per-tenant overrides let you tune these ingestion and retention parameters for specific users or applications without affecting the global cluster.
The most surprising thing about per-tenant overrides is that while you can increase certain limits like max_traces_per_second or trace_idle_period for a tenant, these tenant-specific values are always capped by the global configuration. If your global ingester.max_traces_per_second is set to 10000, you can set a tenant’s override to 20000, but Tempo will still only process 10000 traces per second across the entire cluster. The tenant-specific limit acts as a maximum within the global limit.
The limits.max_traces_per_second in the per-tenant configuration is particularly important. It’s not about how much the ingester can process, but rather a hard cap on how many traces Tempo will accept for that tenant. If the global ingester.max_traces_per_second is 10000 and you set tenant-a.limits.max_traces_per_second to 5000, Tempo will throttle "team-a" at 5000 traces per second, even if the cluster as a whole could handle more. This is crucial for preventing a single noisy tenant from overwhelming the entire system.
Furthermore, trace_idle_period controls how long a trace is considered "active" before it’s considered complete and ready for compaction. A longer trace_idle_period means traces will stay in memory longer, potentially increasing memory usage but also allowing for more complete traces to be captured if spans arrive out of order. Conversely, a shorter period leads to faster compaction but might result in fragmented traces.
The max_block_duration in the global configuration dictates the maximum time a single trace block can span. This helps manage the size of individual blocks, impacting compaction and query performance. Tenant overrides don’t directly change this; they operate within the boundaries set by this global parameter.
The distributor component is where the tenant identification and routing primarily happens, often based on metadata within the trace itself (like the tenant_id header in OpenTelemetry). It directs incoming traces to the appropriate ingester based on these tenant configurations.
The "trick" to understanding per-tenant overrides is realizing that the global settings are the absolute ceiling. You can’t configure a tenant to ingest more traces than the entire cluster is provisioned for, nor can you set a tenant’s trace_idle_period to be indefinitely long if the global max_block_duration is small, as blocks need to be finalized and written. The overrides are about partitioning the global capacity and retention policies, not creating entirely new, independent capabilities.
The next step you’ll encounter is managing the configuration of the distributor and ingester components themselves, as they are the primary actors in enforcing these per-tenant settings dynamically.