Here I am again with the same error, but a different resolution.
ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=group1, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK
In the original post (http://www.givemeit.com/Splunk-Cluster-Indexers-ERROR-IndexerDiscoveryH…), the error message was related to ports 9997, etc. However, I recently replaced and synchronized all of the splunk.secret files and restarted the cluster peers, cluster master, deployment server, and main search head. I started getting a similar message. My main heavy forwarders started showing the error again. Not right away, but eventually after I made a deploy change and the heavy forwarders restarted - I got the dreaded ERROR IndexerDiscoveryHeartbeatThread .
I had Splunk support verify my configuration was still correct, and the last effort was to do a rolling restart on the indexer cluster. It worked! It worked is really a relief and a level of frustration because I had restarted all of the roles individually a few times, but with no success.
From the Cluster Master, I executed the following commands:
[root@clustermaster bin]# ./splunk edit cluster-config -percent_peers_to_restart 20 |
[root@clustermaster bin]# ./splunk rolling-restart cluster-peers |
Reference: https://docs.splunk.com/Documentation/Splunk/6.5.3/Indexer/Userollingre…