OpenTelemetry span attributes are the primary mechanism for enriching trace data with context, but the convention for naming and structuring these attributes is what truly unlocks their power for analysis.
Let’s see this in action. Imagine a request comes into a web service. By default, OpenTelemetry might generate a span with attributes like:
{
"trace_id": "a1b2c3d4e5f678901234567890abcdef",
"span_id": "fedcba9876543210fedcba9876543210",
"name": "HTTP GET /users",
"kind": "SERVER",
"start_time": "2023-10-27T10:00:00Z",
"end_time": "2023-10-27T10:00:01Z",
"attributes": {
"http.method": "GET",
"http.url": "/users",
"http.status_code": 200
}
}
This is okay, but not incredibly useful for querying. If we apply the OpenTelemetry semantic conventions, specifically for HTTP, we get something much richer:
{
"trace_id": "a1b2c3d4e5f678901234567890abcdef",
"span_id": "fedcba9876543210fedcba9876543210",
"name": "GET /users", // More descriptive name
"kind": "SERVER",
"start_time": "2023-10-27T10:00:00Z",
"end_time": "2023-10-27T10:00:01Z",
"attributes": {
"http.request.method": "GET", // Convention: http.request.method
"url.full": "http://example.com/users", // Convention: url.full
"http.response.status_code": 200, // Convention: http.response.status_code
"server.address": "example.com", // Convention: server.address
"server.port": 80, // Convention: server.port
"client.address": "192.168.1.100", // Convention: client.address
"client.port": 54321, // Convention: client.port
"user_agent.original": "Mozilla/5.0 ...", // Convention: user_agent.original
"net.protocol.version": "1.1" // Convention: net.protocol.version
}
}
Notice how the attribute names have changed to follow established patterns. The http.request.method is more specific than http.method. url.full captures the entire URL, not just the path. We’ve also added network and client details.
The problem these conventions solve is fragmentation. Without them, every service, every library, every developer might invent their own way to record the same information. One service might log "user_id", another "userId", another "uid". Tempo, or any other tracing backend, would struggle to correlate these. You’d end up with separate, unjoinable datasets for what is fundamentally the same piece of context.
The OpenTelemetry semantic conventions provide a unified language for describing common operations and entities within a distributed system. They cover a vast range of areas, including:
- HTTP:
http.request.method,http.response.status_code,url.path,url.query - Databases:
db.system,db.name,db.operation,db.user - Messaging:
messaging.system,messaging.destination.name,messaging.operation - RPC:
rpc.system,rpc.method,rpc.service - Cloud Providers:
cloud.provider,cloud.region,cloud.account.id - General:
service.name,process.pid,host.name,user.id
By adhering to these conventions, you ensure that when a request flows through multiple services, the relevant context (like user.id or http.request.method) is consistently attached to spans across the entire trace. This allows you to:
- Filter and Search: Quickly find all traces related to a specific user ID, or all requests to a particular API endpoint.
- Aggregate and Analyze: Group traces by service, method, or status code to identify performance bottlenecks or error patterns.
- Visualize Dependencies: Understand how services interact based on shared attributes.
The magic of conventions isn’t just in the naming; it’s in the agreement. When everyone agrees to use db.system for the database type, regardless of whether it’s PostgreSQL, MySQL, or MongoDB, your queries like db.system = "postgresql" will work universally. This standardization is what makes a distributed tracing system like Tempo truly powerful for debugging and understanding complex applications.
The underlying implementation of these conventions typically involves using libraries or SDKs that are aware of these standards. For example, in Go, you might use otelcontribcol or specific instrumentation libraries that automatically apply these attributes based on the libraries you’re using (like net/http or database/sql). You can also manually add attributes:
import (
"go.opentelemetry.io/otel/attribute"
semconv "go.opentelemetry.io/otel/semconv/v1.17.0" // Use the latest version
)
// ... inside your handler
tp.SpanFromContext(ctx).SetAttributes(
semconv.HTTPRequestMethodKey.String("GET"),
semconv.HTTPResponseStatusCodeKey.Int(200),
semconv.URLFullPath("/users"),
semconv.UserAgentOriginal("MyAwesomeClient/1.0"),
)
The attributes map to specific keys defined in the semantic conventions package. Using semconv.HTTPRequestMethodKey (which resolves to "http.request.method") ensures you’re using the correct, standardized key.
A common pitfall is to assume that all attributes are equally queryable. While Tempo can ingest and store any attribute, the performance and ease of querying are directly tied to how well these attributes align with common analytical patterns. Attributes used for filtering and aggregation, particularly those that are low-cardinality (meaning they have a limited number of distinct values, like status codes or HTTP methods), are indexed for fast lookups. High-cardinality attributes (like full URLs with query parameters or trace IDs) are generally not indexed for general searching but are still essential for individual trace inspection.
When you start instrumenting new services or libraries, consult the OpenTelemetry Semantic Conventions documentation. This is your canonical reference for what attributes exist and how to use them. The conventions are a living document, regularly updated to reflect new technologies and emerging best practices.
The next logical step after mastering span attributes is understanding how to correlate logs and metrics with your traces, often using trace IDs as the common key.