Jaeger ingestion into Tempo is failing because the Tempo Jaeger receiver isn’t properly configured to accept traces via gRPC or Thrift protocols.

Common Causes and Fixes

  1. Incorrect gRPC Port Configuration:

    • Diagnosis: Check your tempo.yaml configuration file for the traces.receiver.jaeger section. Look for the grpc sub-section and its endpoint setting. If it’s not set or incorrectly set, this is the likely culprit.
      grep -A 5 "traces.receiver.jaeger" /etc/tempo/tempo.yaml
      
    • Fix: Ensure the grpc.endpoint is correctly specified and listening on a port that Jaeger agents/collectors are configured to send to. A common default is 14250.
      traces:
        receiver:
          jaeger:
            grpc:
              endpoint: "0.0.0.0:14250"
            thrift_http:
              endpoint: "0.0.0.0:14268"
      
    • Why it works: Tempo needs to explicitly bind to an IP address and port for gRPC traffic. If this isn’t configured, it won’t listen for incoming Jaeger gRPC requests.
  2. Incorrect Thrift HTTP Port Configuration:

    • Diagnosis: Similar to gRPC, examine the thrift_http sub-section within traces.receiver.jaeger in your tempo.yaml.
      grep -A 5 "traces.receiver.jaeger" /etc/tempo/tempo.yaml
      
    • Fix: Verify the thrift_http.endpoint is correctly set, often to 0.0.0.0:14268 for Thrift HTTP.
      traces:
        receiver:
          jaeger:
            grpc:
              endpoint: "0.0.0.0:14250"
            thrift_http:
              endpoint: "0.0.0.0:14268"
      
    • Why it works: Jaeger clients can use Thrift over HTTP as an alternative transport. Tempo must be configured to listen on the specified Thrift HTTP port to receive these traces.
  3. Jaeger Agent/Collector Not Sending to Tempo:

    • Diagnosis: On your Jaeger agent or collector instances, check their configuration (e.g., jaeger-agent.json, jaeger-collector.yaml) to see where they are configured to send traces. If Tempo’s IP and port aren’t listed, this is the issue.
      # Example: Check Jaeger agent config for remote storage
      cat /etc/jaeger/jaeger-agent.json | grep "collector.host"
      
    • Fix: Update the Jaeger agent/collector configuration to point to your Tempo instance’s gRPC or Thrift HTTP endpoint. For example, if Tempo is at 192.168.1.100 and listening on 14250 for gRPC:
      // jaeger-agent.json snippet
      "collector": {
        "host": "192.168.1.100",
        "port": 14250
      }
      
      Then restart the Jaeger agent/collector.
    • Why it works: The Jaeger components are the source of the traces. If they are configured to send traces to the wrong destination (or nowhere), Tempo will never receive them.
  4. Network Firewall Blocking Ports:

    • Diagnosis: Use telnet or nc from the Jaeger agent/collector machine to test connectivity to the Tempo machine’s configured gRPC (14250) and Thrift HTTP (14268) ports.
      # From Jaeger agent/collector machine
      telnet <tempo-ip> 14250
      telnet <tempo-ip> 14268
      
      If these connections time out or are refused, a firewall is likely blocking traffic.
    • Fix: Open the necessary ports (e.g., 14250 TCP, 14268 TCP) in your network firewall or security groups to allow traffic from your Jaeger agents/collectors to your Tempo instance.
    • Why it works: Even if Tempo is listening and Jaeger is sending, network intermediaries can prevent the packets from reaching their destination.
  5. Incorrect Protocol Specified in Jaeger Client:

    • Diagnosis: If you’re instrumenting applications directly with Jaeger clients (not using an agent/collector as a sidecar), examine your application’s Jaeger client configuration. It might be explicitly set to use a different protocol or endpoint.
    • Fix: Ensure your Jaeger client library is configured to use either gRPC or Thrift HTTP and is targeting the correct Tempo endpoint. For example, in Go:
      // Using gRPC
      reporter, err := jaeger.NewRemoteReporter(
          jaeger.Sender(
              grpc.NewSender(
                  grpc.WithEndpoint("192.168.1.100:14250"),
                  grpc.WithTransport(grpc.DefaultTransport()),
              ),
          ),
      )
      // Or using Thrift HTTP
      reporter, err := jaeger.NewRemoteReporter(
          jaeger.Sender(
              http.NewSender(
                  http.WithEndpoint("192.168.1.100:14268"),
              ),
          ),
      )
      
      Recompile and redeploy your application.
    • Why it works: The client library needs to know how and where to send its traces. Mismatched protocol or endpoint configurations will lead to ingestion failures.
  6. Tempo Receiver Not Enabled or Disabled:

    • Diagnosis: In tempo.yaml, ensure the traces.receiver.jaeger block is present and not commented out. Also, check that other receivers (like OTLP) aren’t unintentionally consuming the ports if misconfigured.
      grep "traces.receiver.jaeger" /etc/tempo/tempo.yaml
      
    • Fix: Uncomment or add the traces.receiver.jaeger section if it’s missing or commented out. Ensure no other receiver is configured to use 14250 or 14268 if you intend for Jaeger to use them.
      # Ensure this block is present and uncommented
      traces:
        receiver:
          jaeger:
            grpc:
              endpoint: "0.0.0.0:14250"
            thrift_http:
              endpoint: "0.0.0.0:14268"
      
    • Why it works: Tempo is modular. If a receiver isn’t explicitly enabled in the configuration, it simply won’t start, and Tempo won’t listen on those ports for that protocol.

After fixing these, you might encounter a "context deadline exceeded" error if your underlying storage (like object storage) is slow to respond or misconfigured, preventing Tempo from persisting the ingested traces.

Want structured learning?

Take the full Tempo course →