powered by Jive Software

Cloudera Manager and Spark Cluster via Puppet

I have following prerequisite: I have to arrangement both Cloudera Manager and Spark Cluster by means of Puppet but such that I require insignificant (or none) design through Cloudera Manager UI a while later. Perfect situation that I’m searching for is following:

Topology: 3 hubs (where node1 is spark-master and node2 and node3 are spark-workers)

  • Arrangement start bunch (this fills in not surprisingly) and I have working CDH5.5 Spark group (confirmed by running Spark Pi precedent)

  • Introduce CM server on spark-master node

  • Introduce CM operator on all hubs

  • Begin CM server and operators

I’m using razorsedge/cloudera puppet module to provision Cloudera Manager (https://forge.puppetlabs.com/razorsedge/cloudera) and I have custom made Spark puppet module which support CDH5.5 Spark installation

I’m utilizing razorsedge/cloud

When I open Cloudera Manager UI, I can see every one of the three hubs however I don’t perceive any Spark related details on CM UI dashboard.

When exploring cm operator and server logs, these are the discoveries:

cm operator sign on start ace (was not associated with CM server and can’t be seen on CM UI dashboard)

[12/Jan/2016 23:13:11 +0000] 4678 MainThread agent ERROR Heartbeating to EC2_PUBLIC_DNS:7182 failed

  1. cm agent log on spark-workers (connected to CM server successfully and can be seen on CM UI dashboard)
  2. cm server log on spark-master:

org.apache.avro.AvroRuntimeException: Unknown datum type: java.lang.IllegalArgumentException: Hostname invalid EC2_LOCAL_IPV4

Any idea what might be the issue here?


Wrong forum. Spark here is an instant messenger app. Not Spark Cluster.