Pipelining in Valkey doesn’t actually send commands in parallel; it sends them in a single network round trip, and Valkey processes them sequentially.
Let’s see it in action. Imagine you’ve got a ton of SET operations to perform.
import valkey
# Connect to Valkey
r = valkey.Redis(decode_responses=True)
# Create a pipeline
pipe = r.pipeline()
# Queue up commands
pipe.set('user:1', 'Alice')
pipe.set('user:2', 'Bob')
pipe.set('user:3', 'Charlie')
# Execute the pipeline
results = pipe.execute()
print(results)
This will output [True, True, True], indicating all SET commands were successful. Notice how we queued them up first and then executed once.
The core problem pipelining solves is network latency. Each individual Valkey command involves a round trip: your client sends the command, Valkey processes it, and sends the response back. For a large number of small commands, this latency adds up significantly, becoming the bottleneck for throughput. Pipelining collapses these multiple round trips into a single one.
Here’s how it works internally. When you create a pipeline object, you’re essentially creating a buffer on the client side. You add commands to this buffer using methods like set(), get(), incr(), etc. These commands aren’t sent to the Valkey server yet. They’re just being collected.
When you call execute(), the client library formats all the queued commands into a single request, often separated by newline characters (\r\n). This entire block of text is then sent over the network to the Valkey server in one go.
The Valkey server receives this single request. It then parses the request, executing each command sequentially, one after another, as if they were individual commands. Crucially, it does not process them in parallel. It buffers the responses for each command internally. Once all commands in the batch are processed, it sends all the collected responses back to the client in a single network response.
The client library then receives this single response, parses it, and unpacks the individual results, returning them to you as a list, in the same order the commands were originally queued.
This mechanism dramatically reduces the overhead associated with network communication. Instead of N network round trips for N commands, you have just one. This is why pipelining is a cornerstone for achieving high throughput in Valkey applications, especially when performing many small, independent operations.
You can also chain commands and get their results. For instance, r.pipeline().incr('counter').incr('counter').get('counter').execute() would return [1, 2, 2] if the counter started at 0. The incr calls are executed first, updating the counter, and then the get retrieves the final value.
A common misconception is that WATCH and MULTI/EXEC are the same as pipelining. While they also involve sending multiple commands at once, WATCH/MULTI/EXEC are designed for atomic transactions. WATCH monitors keys for changes, and if any watched key is modified before EXEC, the entire transaction is aborted. Pipelining, on the other hand, offers no such atomicity guarantees. Commands in a pipeline are executed independently, and if one fails, others will still proceed.
The transaction method on a pipeline object is what enables atomic transactions. r.transaction().set('key1', 'val1').get('key1').execute() will ensure that if key1 is modified between the WATCH (implicitly started by transaction) and EXEC, the whole sequence is retried or fails.
The next hurdle you’ll often encounter is managing pipeline size. Sending too many commands in a single pipeline can lead to excessive memory consumption on both the client and server, and can increase the latency of the entire batch if even one command is slow.