Splunk’s KV Store can feel like a black box, but it’s actually just a specialized database designed to hold structured data for fast lookups directly within your Splunk searches.
Let’s see it in action. Imagine you have a list of critical servers and their assigned on-call engineers, and you want to quickly see which engineer is responsible for a server experiencing an alert.
First, we need to create a KV Store collection. You do this via the Splunk UI: Settings > Knowledge > KV Store Collections > New Collection. Let’s call our collection oncall_shifts. We’ll define two fields: server_name (string) and engineer_email (string).
Now, we need to populate it. You can do this through the UI by clicking "New Record" for your oncall_shifts collection and entering data like this:
| server_name | engineer_email |
|---|---|
| webserver-01.prod | alice@example.com |
| dbserver-02.prod | bob@example.com |
| appserver-03.prod | charlie@example.com |
Alternatively, and more commonly for larger datasets, you can use Splunk’s inputlookup and outputlookup commands. Suppose you have a CSV file named oncall_data.csv with the same columns:
server_name,engineer_email
webserver-01.prod,alice@example.com
dbserver-02.prod,bob@example.com
appserver-03.prod,charlie@example.com
You can upload this CSV to Splunk (e.g., to $SPLUNK_HOME/etc/apps/your_app/lookups/) and then run:
| inputlookup oncall_data.csv | outputlookup oncall_shifts
This command reads the CSV and writes its contents into the oncall_shifts KV Store collection.
Now, let’s say you have an alert that includes the server_name. You can join this alert data with your KV Store to retrieve the engineer’s email:
index=production_alerts sourcetype=my_app_logs |
rename server_name as server_name_from_alert |
join server_name_from_alert [| inputlookup oncall_shifts | rename server_name as server_name_from_alert | fields server_name_from_alert, engineer_email] |
table _time, server_name_from_alert, engineer_email, message
In this search:
index=production_alerts sourcetype=my_app_logs: This finds your alert events.rename server_name as server_name_from_alert: We rename theserver_namefield from the alert to avoid naming conflicts with the KV Store field.join server_name_from_alert [...]: This is the core. We join the alert events with the results of a subsearch.| inputlookup oncall_shifts | rename server_name as server_name_from_alert | fields server_name_from_alert, engineer_email: The subsearch reads from ouroncall_shiftsKV Store collection, renames itsserver_namefield to match the renamed field from the alert, and only keeps the fields needed for the join and the final output.table _time, server_name_from_alert, engineer_email, message: Finally, we display the relevant information.
The power here is that the inputlookup from a KV Store collection is highly optimized by Splunk. It doesn’t scan through all your indexed data; it performs a direct lookup against the KV Store’s internal structure, making it much faster than a traditional join against another index.
The real magic of KV Stores lies in their ability to act as dynamic lookup tables that you can update without re-indexing your data. This is crucial for managing frequently changing information like user permissions, IP address classifications, or, as in our example, on-call rotations. Instead of creating and updating large CSV lookup files, you can manage this data within Splunk itself.
When you use outputlookup to write to a KV Store collection, Splunk doesn’t just append data; it performs an upsert (update or insert). If a record with the same _key (or a uniquely indexed field if you’ve configured one) already exists, it’s updated. If not, it’s inserted. This makes it incredibly efficient for keeping your lookup data synchronized.
One common misconception is that KV Stores are just fancy CSVs. While they can be populated from CSVs, their internal storage and retrieval mechanisms are fundamentally different and optimized for fast, direct access. This is why searches using inputlookup on KV Store collections are so performant.
Beyond simple lookups, KV Stores support complex queries using their own SPL-like syntax, including filtering, aggregation, and even nested lookups. You can also define indexes on specific fields within a KV Store collection to further accelerate searches that filter on those fields.
If you’re dealing with data that changes frequently and needs to be queried alongside your indexed events, KV Stores are your go-to solution.
The next step after mastering KV Store lookups is to explore their use in creating dynamic dashboards and managing configurations for Splunk apps.