Splunk Cloud Hybrid Forwarding is a way to get data from your on-premises Splunk Enterprise instances into your Splunk Cloud instance.
Here’s how it works with a practical example. Imagine you have sensitive data on-premises that you want to analyze in Splunk Cloud but can’t send directly. You’d set up a Hybrid Forwarder.
First, on your Splunk Cloud instance, you need to create a "Token-based HTTP Event Collector" (HEC). Navigate to Settings > Data Inputs > HTTP Event Collector. Click New Token. Give it a name, like onprem_to_cloud_forwarder. Under Source type, select _json or a specific type if you know it. For App, choose main or where you want the data to land. Under Advanced Settings, set Token Security to challenge for better security. Click Next and then Submit. You’ll be given a token value, something like abcdef1234567890. Copy this token; you’ll need it.
Now, on your on-premises Splunk Enterprise instance, you’ll configure a Universal Forwarder (or a Heavy Forwarder acting as one) to send data to this HEC. You’ll add an http_event_collector.conf file in $SPLUNK_HOME/etc/system/local/.
[http://onprem_to_cloud_forwarder]
token = abcdef1234567890
host = input-<your_splunk_cloud_instance_id>.cloud.splunk.com
port = 8088
protocol = https
send_raw = true
sourcetype = _json
index = main
Replace <your_splunk_cloud_instance_id> with your actual Splunk Cloud instance ID. You can find this in your Splunk Cloud URL, e.g., https://input-prd-us-west-2a.cloud.splunk.com/.
After saving this file, restart your on-premises Splunk forwarder. Now, any data that this forwarder indexes will be sent to your Splunk Cloud HEC. For example, if you configure it to monitor a file:
# On your on-premises Splunk forwarder
$SPLUNK_HOME/bin/splunk add monitor /opt/app/logs/sensitive_data.log -index main -sourcetype _json
The _json sourcetype tells the HEC to parse the incoming data as JSON. If your data isn’t JSON, you might need to adjust send_raw and sourcetype accordingly, or use a transform in your HEC configuration.
The primary problem this solves is data residency and security. You can keep sensitive data on-premises while still leveraging the scalability, advanced analytics, and managed services of Splunk Cloud. The HEC acts as a secure, authenticated endpoint for incoming data, and the token and challenge settings ensure that only authorized forwarders can send data.
The surprising truth about this setup is that the "forwarder" in "Hybrid Forwarder" is often a misnomer. It’s not necessarily a separate Splunk instance; it’s just a configured Universal Forwarder or a Heavy Forwarder acting as one. The core functionality is the HEC endpoint on Splunk Cloud and the forwarder’s ability to send data to it.
The one thing most people don’t realize is how granularly you can control the data ingestion. You can use different HEC tokens for different on-premises sources, each pointing to different indexes or sourcetypes in Splunk Cloud. This allows for sophisticated data routing and access control without complex network configurations or VPNs.
The next hurdle you’ll likely encounter is optimizing HEC throughput and handling potential network latency issues that could lead to data backlogs.