Vector Tap is how you peek inside your Vector pipelines, letting you see events as they flow through your processing stages in real-time.
Let’s watch it in action. Imagine you’ve got a simple Vector configuration:
[sources.my_source]
type = "generator"
interval = "1s"
probes = 1
[transforms.my_transform]
type = "remap"
inputs = ["my_source"]
source = '''
.message = "Hello, " + .message
'''
[sinks.my_sink]
type = "blackhole"
inputs = ["my_transform"]
This setup generates a message every second, prepends "Hello, " to it, and then discards it. To inspect this, you’d start Vector with vector --tap:
vector --tap
Now, in a separate terminal, you’d tell vector --tap what you’re interested in. If you want to see everything coming out of my_source, you’d run:
vector --tap my_source
You’ll immediately see output like this, appearing as soon as the generator source emits an event:
{"timestamp":"2023-10-27T10:00:00.123Z","message":"probe-0"}
If you want to see what’s happening after the my_transform stage, you’d tap that:
vector --tap my_transform
And the output would now show the transformed message:
{"timestamp":"2023-10-27T10:00:01.123Z","message":"Hello, probe-1"}
You can even tap multiple stages by listing them:
vector --tap my_source my_transform
This gives you a powerful way to debug, understand performance, and verify transformations without needing to configure complex sinks just for inspection.
The fundamental problem Vector Tap solves is the "black box" nature of data pipelines. When an event goes missing or is malformed, you often don’t know where in the chain it went wrong. Tap provides a direct conduit to observe the data at any specified point, shedding light on the internal state of your pipeline. It’s like having a stethoscope for your data flows.
Internally, when you run vector --tap <stage_name>, Vector creates a temporary, in-memory "tap" listener associated with the specified stage. This tap intercepts events after they’ve been processed by that stage but before they are passed to its downstream stages or sinks. The tap then serializes these events (typically to JSON) and prints them to standard output. Crucially, this tapping process is designed to have minimal performance impact, as it doesn’t involve disk I/O or network hops for the tapped events themselves.
The exact levers you control are the names of the stages you wish to tap. These correspond directly to the keys in your Vector configuration file that define sources, transforms, and sinks. You can specify a single stage, or multiple stages separated by spaces. The output format is consistent, making it easy to pipe the tap output to other command-line tools like grep, jq, or awk for more sophisticated analysis.
A common misconception is that tapping an event consumes it or modifies its path. This is not the case. Tapping is a purely observational mechanism. The event is still passed along to its intended destination(s) after being observed by the tap. This non-intrusive nature is key to its utility during live debugging.
The most surprising true thing about Vector Tap is that it can be used to precisely measure the latency introduced by individual transforms. By tapping the output of a source and the output of a transform immediately following it, and then using tools like awk to compare timestamps, you can isolate the processing time of that specific transform in milliseconds.
The next concept you’ll want to explore is how to filter tap output based on event content.