I have following prerequisite: I have to arrangement both Cloudera Manager and Spark Cluster by means of Puppet but such that I require insignificant (or none) design through Cloudera Manager UI a while later. Perfect situation that I’m searching for is following:
Topology: 3 hubs (where node1 is spark-master and node2 and node3 are spark-workers)
-
Arrangement start bunch (this fills in not surprisingly) and I have working CDH5.5 Spark group (confirmed by running Spark Pi precedent)
-
Introduce CM server on spark-master node
-
Introduce CM operator on all hubs
-
Begin CM server and operators
I’m using razorsedge/cloudera puppet module to provision Cloudera Manager (https://forge.puppetlabs.com/razorsedge/cloudera) and I have custom made Spark puppet module which support CDH5.5 Spark installation
I’m utilizing razorsedge/cloud
When I open Cloudera Manager UI, I can see every one of the three hubs however I don’t perceive any Spark related details on CM UI dashboard.
When exploring cm operator and server logs, these are the discoveries:
cm operator sign on start ace (was not associated with CM server and can’t be seen on CM UI dashboard)
[12/Jan/2016 23:13:11 +0000] 4678 MainThread agent ERROR Heartbeating to EC2_PUBLIC_DNS:7182 failed
- cm agent log on spark-workers (connected to CM server successfully and can be seen on CM UI dashboard)
- cm server log on spark-master:
org.apache.avro.AvroRuntimeException: Unknown datum type: java.lang.IllegalArgumentException: Hostname invalid EC2_LOCAL_IPV4
Any idea what might be the issue here?
Thanks,
Sailaja.