The Splunk Deployment Server is the central nervous system for managing your Splunk Universal Forwarders, allowing you to push configuration changes and applications to them en masse.
Let’s see it in action. Imagine you’ve just developed a new custom application, my_custom_app, that you want to deploy to all your forwarders. This app might contain inputs.conf stanzas to monitor new log files, props.conf and transforms.conf for parsing, and perhaps even a dashboard.
First, you’d bundle your application into a .spl file. This is just a standard directory structure for Splunk apps, typically containing default/, local/, bin/, README/, etc.
On your Deployment Server, you’d place this .spl file into the $SPLUNK_HOME/etc/deployment-apps/ directory. For our example, you’d create a directory named my_custom_app inside deployment-apps and put all your app’s files and subdirectories there.
# On the Deployment Server
cd $SPLUNK_HOME/etc/deployment-apps/
mkdir my_custom_app
# Copy all your app's contents into this directory
cp -r /path/to/your/local/my_custom_app/* ./my_custom_app/
Now, you need to tell the Deployment Server which forwarders should receive this app. This is done through a serverclass.conf file, which resides in $SPLUNK_HOME/etc/system/local/ on the Deployment Server.
Here’s an example serverclass.conf:
# On the Deployment Server
# $SPLUNK_HOME/etc/system/local/serverclass.conf
[serverClass:my_app_deploy]
# This is the app we want to deploy
# It must match the directory name in deployment-apps
whitelist.apps = my_custom_app
# This defines which clients (forwarders) belong to this server class
# You can use regex for client names or IP addresses
stateOnClient = enabled
# Example: Deploy to all clients whose hostname starts with 'uf-'
clientgroupRegex = ^uf-
# Alternatively, deploy to specific clients by name
# clients = uf-server-01, uf-server-02
# You can also target clients based on their Splunk version, OS, etc.
# splunk_version = 7.2.1
# os = Linux
After creating or modifying serverclass.conf, you need to restart the Deployment Server Splunk instance for the changes to take effect.
# On the Deployment Server
$SPLUNK_HOME/bin/splunk restart
On the Universal Forwarders, you need to configure them to point to the Deployment Server. This is done in their deploymentclient.conf file, typically located at $SPLUNK_HOME/etc/system/local/ on each forwarder.
# On each Universal Forwarder
# $SPLUNK_HOME/etc/system/local/deploymentclient.conf
[deploymentClient]
# The IP address or hostname of your Deployment Server
# If you have multiple, list them separated by commas
# The server that responds first will be used.
phoneHomeInterval = 60
# The interval in seconds for the forwarder to check in with the deployment server.
# If you have multiple deployment servers, list them like this:
# server = ds1.example.com:8089, ds2.example.com:8089
server = your_deployment_server_ip:8089
The phoneHomeInterval is crucial. It dictates how often the forwarder will "phone home" to the Deployment Server to check for new configurations or apps. A common value is 60 seconds, but you might decrease it for faster deployments or increase it to reduce load on the Deployment Server.
Once the forwarder checks in, the Deployment Server will see that the forwarder matches the clientgroupRegex (or specific clients) in serverclass.conf for the my_app_deploy server class. It will then push the my_custom_app to the forwarder.
The forwarder receives the app and places it into its own $SPLUNK_HOME/etc/apps/ directory. It will then automatically reload its configuration, including the new inputs, props, and transforms from your my_custom_app. You don’t need to restart the forwarder.
You can verify the deployment from the Deployment Server’s perspective by navigating to Settings > Forwarder Management in the Splunk Web UI. You should see your forwarders listed, and under their "Status" or "Apps" column, you’ll see my_custom_app listed as deployed.
On a forwarder, you can check its deployment status by looking at the splunkd.log file, usually located in $SPLUNK_HOME/var/log/splunk/splunkd.log. You’ll see messages indicating it’s connecting to the deployment server and receiving apps.
The true power here is in managing configurations dynamically. If you need to update my_custom_app, you simply update the files in the deployment-apps/my_custom_app directory on the Deployment Server, restart the Deployment Server, and all targeted forwarders will automatically pull down the updated version on their next phoneHomeInterval. This eliminates the need to SSH into each forwarder and manually copy files.
One subtle but powerful aspect is the stateOnClient directive. When set to enabled, as in our example, the Deployment Server ensures that the specified apps are present on the client and that any local modifications to those apps on the client are preserved. If you were to set it to disabled, the Deployment Server would remove the app from the client. If you were to set stateOnClient = force, the Deployment Server would overwrite any local changes on the forwarder with the version from the Deployment Server, effectively making the deployed app read-only from the forwarder’s perspective.
The next step in mastering deployments is understanding how to manage different configurations for different groups of forwarders using multiple server classes, or how to use deployment-apps to push configuration bundles that don’t contain full applications, but rather specific stanzas for inputs.conf, outputs.conf, etc.