Splunk Cluster Indexers ERROR IndexerDiscoveryHeartbeatThread

I noticed I wasn't receiving all of the log data I was expecting from Splunk Heavy Forwarders to my newly setup Splunk Index Cluster. It was a simple problem, but it was very difficult to figure out based on the log messages.

This log file will indicate there is an actual credential problem. Most likely the passSymm4key value is wrong.
01-11-2017 17:10:03.017 -0500 ERROR IndexerDiscoveryHeartbeatThread - failed heartbeat for group=group1 uri=https://yourclustermanager:8089/services/indexer_discovery http_response=Unauthorized

However, this log indicates (not clearly at all), that the cluster peers are not listening on 9997. This can be a configuration issue with firewalld or inputs.conf

01-11-2017 19:48:54.027 -0500 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 2040 seconds.
01-11-2017 19:48:58.642 -0500 ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=group1, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK

Verify that /opt/splunk/etc/system/local/inputs.conf has either splunktcp or splunk-tcpssl (not both) below the host = value.

[splunktcp://9997]
disabled = 0
Verify that firewalld is open to 9997/tcp.
firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: em1
sources:
services: dhcpv6-client ssh
ports: 8089/tcp 9997/tcp 8000/tcp 8080/tcp
protocols:
masquerade: no
forward-ports:
sourceports:
icmp-blocks:
rich rules:

If not, add the port permanently:

firewall-cmd --permanent --add-port=9997/tcp
firewall-cmd --reload

Update- This may help resolve our issue as well - http://www.givemeit.com/Splunk-Cluster-Indexers-ERROR-IndexerDiscoveryH…