Valkey’s ziplist encoding for hashes is actually a performance regression under certain common access patterns, not always the optimization it’s made out to be.
Let’s see it in action. Imagine a Valkey instance with a hash key myhash storing a few fields.
> SET myhash "initial value"
OK
> HSET myhash field1 "value1" field2 "value2" field3 "value3"
(integer) 3
> HGETALL myhash
1) "field1"
2) "value1"
3) "field2"
4) "value2"
5) "field3"
6) "value3"
By default, Valkey tries to be clever. When a hash has a small number of fields and their total serialized length is below a certain threshold, it uses ziplist. Otherwise, it switches to a standard hash table.
Here’s how you can inspect the encoding:
> OBJECT ENCODING myhash
"ziplist"
This ziplist is a special, space-efficient data structure. It stores elements contiguously in memory, like an array, but with forward and backward pointers to allow iteration. For hashes with few fields and short keys/values, this means less memory overhead per entry and potentially faster access because everything is in one contiguous block, reducing cache misses.
The problem arises when you start accessing these fields. When ziplist is used, HGET, HSET, HDEL, and HEXISTS operations don’t just jump to a memory address. They have to traverse the ziplist from the beginning (or end) to find the specific field.
Consider this:
> HGET myhash field3
"value3"
> OBJECT ENCODING myhash
"ziplist"
Even though we just fetched field3, the encoding remains ziplist. If field3 was the last element, this lookup was relatively quick. But what if it was the Nth element in a ziplist of size M? The lookup time becomes O(N), where N is the position of the field. As the ziplist grows, this linear scan becomes a bottleneck.
Now, let’s add more fields.
> HSET myhash field4 "value4" field5 "value5" field6 "value6" field7 "value7" field8 "value8" field9 "value9" field10 "value10"
(integer) 7
> OBJECT ENCODING myhash
"ziplist"
We’re still at ziplist. Let’s keep going.
> HSET myhash field11 "value11" field12 "value12" field13 "value13" field14 "value14" field15 "value15" field16 "value16" field17 "value17" field18 "value18" field19 "value19" field20 "value20"
(integer) 10
> OBJECT ENCODING myhash
"hashtable"
Ah, now it’s switched! Valkey automatically re-encoded the hash from ziplist to hashtable once it exceeded certain internal thresholds for the number of fields and/or their total size. The hashtable encoding uses a more traditional hash table implementation, similar to what you’d find in many programming languages. Each entry is stored in a bucket, and access is typically O(1) on average.
The thresholds are controlled by two configuration parameters:
hash-max-ziplist-entries: The maximum number of fields a hash can have before it’s converted fromziplisttohashtable. The default is 512.hash-max-ziplist-value: The maximum size (in bytes) of a single field’s value (key + value serialized) before it’s converted fromziplisttohashtable, regardless of the number of entries. The default is 64.
If you have a use case where you know your hashes will often grow beyond, say, 50 fields, but you still want O(1) average access time, you might want to force the hashtable encoding early. You can do this by lowering hash-max-ziplist-entries.
For example, to make Valkey switch to hashtable when a hash exceeds 100 entries, you’d add this to your valkey.conf:
hash-max-ziplist-entries 100
And then restart Valkey. Any new hashes created or existing hashes that grow beyond 100 entries will use hashtable. You can also manually trigger a re-encoding (though this is usually unnecessary as Valkey handles it automatically):
> OBJECT ENCODING myhash
"ziplist"
> OBJECT ENCODING myhash
"hashtable"
The second OBJECT ENCODING command, after the automatic re-encoding has occurred (e.g., by adding enough elements), will reveal the change. There isn’t a direct command to force a re-encode on demand without modifying the data, but Valkey’s automatic conversion is the key mechanism.
The real surprise is that ziplist isn’t always better. While it saves memory for small hashes, its performance characteristics for lookups degrade linearly as entries are added. For frequently accessed or growing hashes, the overhead of ziplist traversal can easily outweigh its memory savings, making the hashtable a more performant choice even if it uses slightly more memory.
The next step is understanding how Valkey handles similar encoding optimizations for other data structures, like list-max-ziplist-entries for lists.