SPLUNK CLUSTER INDEXERS ERROR INDEXERDISCOVERYHEARTBEATTHREAD Part 2

Here I am again with the same error, but a different resolution.

ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=group1, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK

In the original post (http://www.givemeit.com/Splunk-Cluster-Indexers-ERROR-IndexerDiscoveryH…), the error message was related to ports 9997, etc. However, I recently replaced and synchronized all of the splunk.secret files and restarted the cluster peers, cluster master, deployment server, and main search head. I started getting a similar message. My main heavy forwarders started showing the error again. Not right away, but eventually after I made a deploy change and the heavy forwarders restarted - I got the dreaded ERROR IndexerDiscoveryHeartbeatThread .

I had Splunk support verify my configuration was still correct, and the last effort was to do a rolling restart on the indexer cluster. It worked! It worked is really a relief and a level of frustration because I had restarted all of the roles individually a few times, but with no success.

From the Cluster Master, I executed the following commands:

[root@clustermaster bin]# ./splunk edit cluster-config -percent_peers_to_restart 20
The cluster-config property has been edited.

[root@clustermaster bin]# ./splunk rolling-restart cluster-peers
Rolling Restart of all the cluster peers has been kicked off. It might take some time for completion.

Reference: https://docs.splunk.com/Documentation/Splunk/6.5.3/Indexer/Userollingre…