The most surprising thing about log routing with Vector is that it’s not a fixed path; it’s a dynamic, self-healing mesh.
Imagine you have a cluster of services generating logs. You want to collect these logs, process them (filter, enrich, transform), and send them to a central store like Elasticsearch or S3. This is where Vector comes in, acting as your log shipper. But "routing" isn’t just about where logs go; it’s about how they get there and what happens if parts of the journey break.
Let’s look at a typical setup. We have Vector agents running on each host, and a central Vector aggregator (or a cluster of them).
Agent Topology (The Source)
On your application hosts, each Vector agent is configured to watch specific log files or listen on network ports.
# agent.toml
[sources.my_app_logs]
type = "file"
include = ["/var/log/my_app/*.log"]
[sources.another_service_logs]
type = "journald"
These agents will collect logs from /var/log/my_app/ and systemd journald entries.
Aggregator Topology (The Sink)
On your aggregator hosts, Vector is configured to receive logs from the agents and then send them to their final destinations.
# aggregator.toml
[sources.agent_input]
type = "http_client" # Vector agents will POST logs to this
address = "0.0.0.0:8080"
codec = "json"
[sinks.elasticsearch_output]
type = "elasticsearch"
inputs = ["agent_input"] # Process logs from the agent_input source
endpoint = "http://elasticsearch-host:9200"
index = "logs-%Y.%m.%d"
Here, agent_input is a source that listens for incoming HTTP requests (which is how agents send data). elasticsearch_output is a sink that takes logs from agent_input and sends them to Elasticsearch.
The "Routing" Magic: How Agents Find Aggregators
This is where the "topology" and "aggregator discovery" come into play. You don’t manually tell each agent the IP address of every aggregator. That’s brittle. Instead, agents can be configured to discover aggregators dynamically.
One common way is using Vector’s load_balanced sink, often combined with a service discovery mechanism or a static list of endpoints.
Agent Config with Load Balancing:
# agent.toml (modified)
# ... (sources defined above) ...
[sinks.to_aggregator]
type = "http"
# This is the key: a load-balanced endpoint.
# Vector will automatically round-robin or failover between these.
# In a real scenario, this might come from Kubernetes service discovery, Consul, etc.
endpoint = "http://aggregator-service:8080"
inputs = ["my_app_logs", "another_service_logs"]
encoding.codec = "json"
When the agent sends logs to aggregator-service:8080, Vector’s HTTP sink, configured with a load-balanced endpoint, will:
- Discover Aggregators: If
aggregator-serviceresolves to multiple IP addresses (e.g., via DNS or Kubernetes), Vector will maintain a list. - Round-Robin: It will send logs to the aggregators in a round-robin fashion.
- Health Checks & Failover: Crucially, Vector periodically checks the health of the endpoints. If an aggregator becomes unresponsive, the agent will stop sending logs to it and route them to the healthy ones. If the unresponsive aggregator comes back online, Vector will start sending logs to it again.
This dynamic discovery and health-checking is the "self-healing mesh" aspect. If an aggregator node dies, logs are automatically rerouted. If a new aggregator node is added, agents will discover and start using it.
Internal Flow: The Vector Pipeline
Each Vector instance (agent or aggregator) has a pipeline. Logs flow through sources -> transforms (optional) -> sinks.
- Sources: Ingest data (e.g.,
file,journald,http_client). - Transforms: Process data (e.g.,
filter,remap,mutate,kubernetes_enrichment). - Sinks: Export data (e.g.,
elasticsearch,kafka,s3,http).
The inputs directive in a sink or transform specifies which upstream components’ output it consumes. This forms the Directed Acyclic Graph (DAG) of your log processing.
The One Thing Most People Don’t Know: Backpressure and Flow Control
Vector doesn’t just blindly shovel data. If an aggregator is overwhelmed (e.g., its Elasticsearch sink is slow to acknowledge data), it will signal backpressure upstream. This means the http_client source on the agent will slow down its rate of sending data. This prevents agents from buffering infinite amounts of logs in memory or on disk, which could otherwise lead to OOM errors or disk exhaustion. It’s a fundamental mechanism for stability.
If you have agents that can’t reach aggregators, or aggregators that can’t reach their sinks, you’ll see logs drop or queues back up. Understanding this flow control is key to debugging.
The next concept you’ll grapple with is how to perform complex transformations and enrichments within the pipeline itself.