Grafana Tempo can be deployed on Kubernetes using either a DaemonSet or a Sidecar pattern, and understanding their differences is key to optimizing your tracing setup.
Let’s see Tempo in action. Imagine you have a microservices application running on Kubernetes. Each service generates traces, which are then sent to Tempo for aggregation and analysis.
Here’s a simplified example of how traces might flow. A request comes into your frontend service. It calls user-service, which in turn calls order-service. Each of these calls is a "span," and they’re all linked together by a common "trace ID."
frontend (request start)
-> user-service (request start)
-> order-service (request start)
-> order-service (db query)
-> order-service (request end)
-> user-service (request end)
-> frontend (request end)
Tempo’s job is to receive these spans, group them by trace ID, and store them efficiently.
DaemonSet Deployment
In a DaemonSet deployment, a Tempo instance runs on each Kubernetes node. Your application pods, running on those same nodes, send their traces directly to the local Tempo agent.
Why it’s cool: This is great for reducing network hops. Traces stay local to the node, minimizing latency and bandwidth usage between your application pods and the tracing backend. It also provides built-in high availability for trace ingestion at the node level; if one Tempo agent on a node fails, others on different nodes continue to operate.
How it works internally:
- Application Pods: Your application pods are configured to send traces to
localhost:portor a specific node IP. - Tempo DaemonSet Pods: The DaemonSet ensures one Tempo agent pod is running on every worker node. These agents listen for traces on a specific port (e.g., 6831 for Jaeger UDP, 4317 for OTLP gRPC).
- Ingestion: Application pods on a node send traces to their local Tempo agent.
- Storage: The Tempo agents then write these traces to a shared, distributed storage backend (like S3, GCS, or an object storage compatible with MinIO).
Levers you control:
- Resource Allocation: You define CPU and memory requests/limits for the Tempo DaemonSet pods.
- Storage Configuration: You configure the object storage bucket and credentials Tempo uses.
- Ingestion Ports: You can specify which ports Tempo listens on for different tracing protocols (Jaeger, OTLP, Zipkin).
- Node Affinity/Tolerations: You can control which nodes the Tempo DaemonSet pods are scheduled on.
Sidecar Deployment
In a Sidecar deployment, a Tempo agent runs as a separate container within the same pod as your application container.
Why it’s cool: This pattern offers even tighter coupling between your application and its tracing agent. It’s particularly useful in environments where you have strict network policies or want to guarantee that the tracing agent is always co-located with its application. If the application pod is rescheduled, its associated Tempo sidecar moves with it.
How it works internally:
- Application Pod: A pod is defined with at least two containers: your application container and a Tempo agent container.
- Communication: Your application container is configured to send traces to
localhost:portwithin the same pod. - Sidecar Agent: The Tempo agent container in the pod receives these traces.
- Remote Ingestion: The Tempo agent sidecar then forwards these traces to a centralized Tempo backend (which could itself be deployed as a Deployment, not a DaemonSet). This central backend is responsible for writing to the object storage.
Levers you control:
- Resource Allocation: You define CPU and memory for both the application container and the Tempo sidecar container.
- Sidecar Configuration: You specify the Tempo agent configuration within the pod’s YAML, including its endpoint for receiving traces and its upstream endpoint for sending to the central Tempo backend.
- Central Tempo Backend Configuration: You configure the main Tempo deployment’s storage and ingestion.
The One Thing Most People Don’t Know
When using the DaemonSet pattern, the Tempo agents themselves are stateless regarding trace data; they only ingest and immediately forward to your configured object storage. The state (the traces) lives entirely in that external object store. This means you can scale the DaemonSet up or down, or even restart all agents, without losing any historical trace data. The agents are effectively just very fast, local ingestion points.
Next Steps
Once you’ve mastered Tempo’s deployment patterns, you’ll want to explore how to effectively query and visualize those traces using Grafana’s Explore view and build dashboards based on trace data.