The Splunk dispatch directory is a temporary holding area for search results and other job-related data, and it can grow to consume significant disk space if not managed.
Let’s see it in action. Imagine you’ve run a large search in Splunk. Splunk creates a directory in /opt/splunk/var/run/splunk/dispatch to store the intermediate and final results of that search. The directory name is a unique job ID, like 1678886400.12345. Inside, you’ll find files like results.csv.gz, events.json.gz, and various .csv files for different stages of the search. If you run many searches, or searches that produce massive result sets, these directories can stack up.
Here’s how to clean it up:
1. Identify Large Dispatch Directories:
You can find these directories by looking at their modification times and sizes.
sudo find /opt/splunk/var/run/splunk/dispatch/ -type d -mtime +7 -print0 | xargs -0 du -sh
This command finds directories modified more than 7 days ago and shows their total size.
2. Understand the Risk:
Deleting these directories is generally safe, as they contain temporary search artifacts. However, if a search is currently running and its results are stored here, deleting it will interrupt that search.
3. Manual Deletion (with Caution):
For a targeted cleanup, you can manually remove old directories.
sudo rm -rf /opt/splunk/var/run/splunk/dispatch/1678886400.12345
Replace 1678886400.12345 with the actual directory name you want to remove. This surgically removes the specified job’s dispatch directory.
4. Automated Cleanup with Splunk Settings:
Splunk has built-in mechanisms to manage this. The most common setting to adjust is max_dispatch_age.
-
Locate
server.conf: This file is typically located at/opt/splunk/etc/system/local/server.confor within an app’s local directory (e.g.,/opt/splunk/etc/apps/search/local/server.conf). -
Edit
server.conf: Add or modify themax_dispatch_agesetting within the[general]stanza.[general] max_dispatch_age = 24hThis setting tells Splunk to automatically delete dispatch directories older than 24 hours. You can set it to
12hfor 12 hours,3dfor 3 days, etc. -
Restart Splunk: After saving
server.conf, you must restart Splunk for the changes to take effect.sudo /opt/splunk/bin/splunk restart
5. Monitor Disk Usage:
After implementing automated cleanup, monitor your disk space regularly to ensure it’s staying within acceptable limits.
df -h /opt/splunk/var
This command will show you the current disk usage for the partition where your Splunk var directory resides.
6. Consider max_disk_usage:
Another setting to be aware of is max_disk_usage in limits.conf. While max_dispatch_age focuses on time, max_disk_usage sets a hard limit on the total disk space the dispatch directory can consume.
-
Locate
limits.conf: Typically at/opt/splunk/etc/system/local/limits.conf. -
Edit
limits.conf:[dispatch] max_disk_usage = 10GBThis limits the entire dispatch directory to 10GB. When this limit is reached, Splunk will start deleting the oldest dispatch directories, even if they haven’t reached their
max_dispatch_age.
7. Splunk’s Internal Cleanup Process:
Splunk has a background process that periodically checks and cleans up old dispatch directories based on your server.conf and limits.conf settings. You don’t typically need to trigger this manually; it runs automatically.
The most common reason for the dispatch directory growing excessively, beyond what max_dispatch_age can handle, is a misconfiguration or a Splunk restart that doesn’t allow the cleanup process to run to completion. Sometimes, a corrupted job state can prevent a directory from being deleted.
If you continue to see excessive growth, the next area to investigate is Splunk’s internal logging for any errors related to job management or file deletion.