Tempo doesn’t actually store trace data itself; it relies on object storage (like S3, GCS, or MinIO) for that. The retention policy you’re talking about is really about how long your object storage holds onto the trace data, and how you tell Tempo to stop writing new trace data to object storage after a certain point.
Let’s see Tempo in action. Imagine you have a distributed system, and requests are flowing through several microservices. Tempo’s job is to collect the spans from each service that make up a single trace and store them.
Here’s a simplified view of a trace being sent to Tempo:
- Service A receives a request, generates a trace ID, and sends a span. It then calls Service B.
- Service B receives the call, uses the same trace ID, generates its own span, and sends it to Tempo. It might call Service C.
- Service C does the same, sending its span to Tempo.
- Tempo receives these spans and, after a short delay (to ensure all spans for a trace arrive), batches them up and writes them as a single object (e.g., a
.jsonor.snappyfile) to its configured object storage.
This is what the configuration for Tempo’s object storage might look like:
# tempo.yaml
distributor:
receivers:
- jaeger
- zipkin
- otlp
storage:
trace:
backend: s3 # or gcs, azure, minio
s3:
# Your S3 bucket configuration
bucket: "my-tempo-traces-bucket"
endpoint: "s3.amazonaws.com"
region: "us-east-1"
access_key_id: "YOUR_ACCESS_KEY_ID"
secret_access_key: "YOUR_SECRET_ACCESS_KEY"
Now, how do you control how long this data stays around? Tempo itself doesn’t have a "retention" setting in the traditional sense for deleting old data. Instead, it relies on lifecycle policies configured directly on your object storage.
For example, if you’re using AWS S3, you’d configure a lifecycle rule on your my-tempo-traces-bucket. Here’s how you might set it up to delete objects older than 30 days:
- Navigate to your S3 bucket in the AWS Management Console.
- Go to the Management tab.
- Under Lifecycle rules, click Create lifecycle rule.
- Give your rule a name, e.g.,
DeleteTracesAfter30Days. - Choose to apply the rule to All objects in the bucket or a specific prefix (if you organize your traces by date, for instance).
- Under Lifecycle rule actions, select Expire current versions of objects.
- Enter
30in the Days after object creation field. - Save the rule.
S3 will then automatically delete objects from my-tempo-traces-bucket that are older than 30 days. Tempo, when it needs to read a trace, will simply receive a "not found" error for those deleted objects and move on.
The key insight is that Tempo doesn’t manage deletion; it consumes data from storage that is managed by another system. You can influence what Tempo writes by configuring its storage.trace.max_block_duration, which determines how long it waits before flushing a batch of spans to object storage. But for retention, look to your object store’s lifecycle management.
The next thing you’ll likely want to configure is how Tempo scales and distributes the load of receiving and processing traces across multiple instances.