Vector’s Datadog output can send logs and metrics to Datadog, but it’s not just a simple pass-through.
Here’s Vector sending logs and metrics to Datadog in action.
[sources.file_logs]
type = "file"
include = ["/var/log/app.log"]
[transforms.parse_log]
type = "regex"
inputs = ["file_logs"]
regex = "^(?P<timestamp>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d+Z) (?P<level>\\w+) (?P<message>.*)$"
parse_from = "log"
[transforms.remap_level]
type = "remap"
inputs = ["parse_log"]
source = '''
.level = parse_level(.level)
'''
[transforms.collect_metrics]
type = "log_to_metric"
inputs = ["remap_level"]
namespace = "my_app"
// Convert log level to a count metric
// Example: INFO logs become my_app.log_level.info count
// Example: WARN logs become my_app.log_level.warn count
// Example: ERROR logs become my_app.log_level.error count
// This transform will create a metric for each unique value in the '.level' field.
// The value of the metric will be 1 for each log line.
// The metric name will be '<namespace>.log_level.<level_value>'
# The 'group_by' field is essential for creating distinct metrics for each log level.
group_by = ["level"]
drop_original_event = true
[sinks.datadog_logs]
type = "datadog_events"
inputs = ["remap_level"]
api_key = "YOUR_DATADOG_API_KEY"
# Datadog expects logs in a specific format. Vector maps common log fields automatically.
# For custom attributes, you can use the 'extra_attributes' configuration.
# Example: extra_attributes = { "service": "my-web-app", "env": "production" }
[sinks.datadog_metrics]
type = "datadog_metrics"
inputs = ["collect_metrics"]
api_key = "YOUR_DATADOG_API_KEY"
# The 'namespace' defined in the 'log_to_metric' transform is used here.
# Metrics will appear under this namespace in Datadog.
[sinks.datadog_metrics.tags]
# You can add static tags to all metrics sent to Datadog.
service = "my-vector-agent"
version = "1.0.0"
This setup demonstrates Vector’s ability to both ingest raw log files, parse them into structured data, and then transform that structured data into metrics that Datadog can understand. The log_to_metric transform is key here, allowing you to aggregate log events into time-series metrics. The datadog_events sink handles sending processed logs as Datadog events, while datadog_metrics sends the generated metrics.
The core problem Vector solves here is bridging the gap between unstructured or semi-structured log data and the structured, time-series metric data that monitoring platforms like Datadog excel at. It provides a flexible way to extract meaningful signals from logs and turn them into actionable metrics without needing to instrument your application code directly for every metric you want to track. You define the transformation logic in Vector, and it handles the rest.
The mental model for Vector’s Datadog output involves understanding its distinct sinks for logs (datadog_events) and metrics (datadog_metrics). Each sink requires specific configuration related to authentication (API keys) and how data should be mapped to Datadog’s schema. For logs, Vector attempts to map standard fields like message, timestamp, and level automatically. For metrics, the log_to_metric transform is crucial for defining what constitutes a metric and how it’s named and aggregated. You can also enrich both logs and metrics with custom tags and attributes, providing context for filtering and analysis in Datadog.
A subtle but powerful aspect is how Vector handles metric aggregation before sending to Datadog. When you use log_to_metric with group_by, Vector is not just sending every single log line as a metric point. It’s counting occurrences of distinct level values (in this example) within its internal processing windows. This pre-aggregation significantly reduces the volume of data sent to Datadog, saving costs and improving performance, as Datadog’s metric ingestion is often priced per point. You’re effectively telling Vector to do the counting for you.
Next, you’ll want to explore how to use Vector’s datadog_agent sink to leverage the Datadog Agent’s advanced features like trace forwarding and APM.