Jaeger ingestion into Tempo is failing because the Tempo Jaeger receiver isn’t properly configured to accept traces via gRPC or Thrift protocols.
Common Causes and Fixes
-
Incorrect gRPC Port Configuration:
- Diagnosis: Check your
tempo.yamlconfiguration file for thetraces.receiver.jaegersection. Look for thegrpcsub-section and itsendpointsetting. If it’s not set or incorrectly set, this is the likely culprit.grep -A 5 "traces.receiver.jaeger" /etc/tempo/tempo.yaml - Fix: Ensure the
grpc.endpointis correctly specified and listening on a port that Jaeger agents/collectors are configured to send to. A common default is14250.traces: receiver: jaeger: grpc: endpoint: "0.0.0.0:14250" thrift_http: endpoint: "0.0.0.0:14268" - Why it works: Tempo needs to explicitly bind to an IP address and port for gRPC traffic. If this isn’t configured, it won’t listen for incoming Jaeger gRPC requests.
- Diagnosis: Check your
-
Incorrect Thrift HTTP Port Configuration:
- Diagnosis: Similar to gRPC, examine the
thrift_httpsub-section withintraces.receiver.jaegerin yourtempo.yaml.grep -A 5 "traces.receiver.jaeger" /etc/tempo/tempo.yaml - Fix: Verify the
thrift_http.endpointis correctly set, often to0.0.0.0:14268for Thrift HTTP.traces: receiver: jaeger: grpc: endpoint: "0.0.0.0:14250" thrift_http: endpoint: "0.0.0.0:14268" - Why it works: Jaeger clients can use Thrift over HTTP as an alternative transport. Tempo must be configured to listen on the specified Thrift HTTP port to receive these traces.
- Diagnosis: Similar to gRPC, examine the
-
Jaeger Agent/Collector Not Sending to Tempo:
- Diagnosis: On your Jaeger agent or collector instances, check their configuration (e.g.,
jaeger-agent.json,jaeger-collector.yaml) to see where they are configured to send traces. If Tempo’s IP and port aren’t listed, this is the issue.# Example: Check Jaeger agent config for remote storage cat /etc/jaeger/jaeger-agent.json | grep "collector.host" - Fix: Update the Jaeger agent/collector configuration to point to your Tempo instance’s gRPC or Thrift HTTP endpoint. For example, if Tempo is at
192.168.1.100and listening on14250for gRPC:
Then restart the Jaeger agent/collector.// jaeger-agent.json snippet "collector": { "host": "192.168.1.100", "port": 14250 } - Why it works: The Jaeger components are the source of the traces. If they are configured to send traces to the wrong destination (or nowhere), Tempo will never receive them.
- Diagnosis: On your Jaeger agent or collector instances, check their configuration (e.g.,
-
Network Firewall Blocking Ports:
- Diagnosis: Use
telnetorncfrom the Jaeger agent/collector machine to test connectivity to the Tempo machine’s configured gRPC (14250) and Thrift HTTP (14268) ports.
If these connections time out or are refused, a firewall is likely blocking traffic.# From Jaeger agent/collector machine telnet <tempo-ip> 14250 telnet <tempo-ip> 14268 - Fix: Open the necessary ports (e.g., 14250 TCP, 14268 TCP) in your network firewall or security groups to allow traffic from your Jaeger agents/collectors to your Tempo instance.
- Why it works: Even if Tempo is listening and Jaeger is sending, network intermediaries can prevent the packets from reaching their destination.
- Diagnosis: Use
-
Incorrect Protocol Specified in Jaeger Client:
- Diagnosis: If you’re instrumenting applications directly with Jaeger clients (not using an agent/collector as a sidecar), examine your application’s Jaeger client configuration. It might be explicitly set to use a different protocol or endpoint.
- Fix: Ensure your Jaeger client library is configured to use either gRPC or Thrift HTTP and is targeting the correct Tempo endpoint. For example, in Go:
Recompile and redeploy your application.// Using gRPC reporter, err := jaeger.NewRemoteReporter( jaeger.Sender( grpc.NewSender( grpc.WithEndpoint("192.168.1.100:14250"), grpc.WithTransport(grpc.DefaultTransport()), ), ), ) // Or using Thrift HTTP reporter, err := jaeger.NewRemoteReporter( jaeger.Sender( http.NewSender( http.WithEndpoint("192.168.1.100:14268"), ), ), ) - Why it works: The client library needs to know how and where to send its traces. Mismatched protocol or endpoint configurations will lead to ingestion failures.
-
Tempo Receiver Not Enabled or Disabled:
- Diagnosis: In
tempo.yaml, ensure thetraces.receiver.jaegerblock is present and not commented out. Also, check that other receivers (like OTLP) aren’t unintentionally consuming the ports if misconfigured.grep "traces.receiver.jaeger" /etc/tempo/tempo.yaml - Fix: Uncomment or add the
traces.receiver.jaegersection if it’s missing or commented out. Ensure no other receiver is configured to use14250or14268if you intend for Jaeger to use them.# Ensure this block is present and uncommented traces: receiver: jaeger: grpc: endpoint: "0.0.0.0:14250" thrift_http: endpoint: "0.0.0.0:14268" - Why it works: Tempo is modular. If a receiver isn’t explicitly enabled in the configuration, it simply won’t start, and Tempo won’t listen on those ports for that protocol.
- Diagnosis: In
After fixing these, you might encounter a "context deadline exceeded" error if your underlying storage (like object storage) is slow to respond or misconfigured, preventing Tempo from persisting the ingested traces.