Unable to configure Hazelcast clustering

Hi All,

I am trying to setup Hazelcast cluster in Rackspace cloud environment. There are two open fire server chat01 and chat02 behind a load balancer sharing a common db server. I have disabled the firewall for now, and servers can talk to each other on a local interface visible within cloud.

I have installed and configured open fire and hazelcast on both. Now when I see from Openfire Admin Console in the clustering section: My chat01 is listing itself and the other node in cluster over view section and clicking on both of them show details. Chat02 also shows both, but clicking on the other server returns a blank page instead of details. But then it is intermittent, if I keep refreshing the page sometimes it shows details of the other box which is chat01 but sometimes it is blank, whenever I see the error I see following log:

2014.12.06 12:12:46 com.jivesoftware.util.cache.ClusteredCacheFactory - Failed to execute cluster task within 30 seconds

java.util.concurrent.TimeoutException

at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.resolveResponse(Invocati onImpl.java:466)

at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.get(InvocationImpl.java: 314)

at com.hazelcast.util.executor.DelegatingFuture.get(DelegatingFuture.java:66)

at com.jivesoftware.util.cache.ClusteredCacheFactory.doSynchronousClusterTask(Clus teredCacheFactory.java:334)

at org.jivesoftware.util.cache.CacheFactory.doSynchronousClusterTask(CacheFactory. java:586)

at org.jivesoftware.openfire.admin.system_002dclustering_jsp._jspService(system_00 2dclustering_jsp.java:123)

at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:547)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1359)

at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:11 8)

at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.util.LocaleFilter.doFilter(LocaleFilter.java:74)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingF ilter.java:50)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:78)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:164)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:478)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:520)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:22 7)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:94 1)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:409)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186 )

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:875 )

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandler Collection.java:250)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.jav a:149)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)

at org.eclipse.jetty.server.Server.handle(Server.java:349)

at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:441)

at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConne ction.java:919)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:582)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:51 )

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.jav a:586)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java :44)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598 )

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)

at java.lang.Thread.run(Thread.java:745)

Log with debug info enabled:

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)

at java.lang.Thread.run(Thread.java:745)

2014.12.06 12:01:52 org.jivesoftware.openfire.container.PluginServlet -

java.lang.NullPointerException

at org.jivesoftware.openfire.plugin.hazelcast.system_002dclustering_002dnode_jsp._ jspService(system_002dclustering_002dnode_jsp.java:159)

at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

at org.jivesoftware.openfire.container.PluginServlet.handleJSP(PluginServlet.java: 294)

at org.jivesoftware.openfire.container.PluginServlet.service(PluginServlet.java:10 7)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:547)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1359)

at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:78)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:164)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:11 8)

at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.util.LocaleFilter.doFilter(LocaleFilter.java:74)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingF ilter.java:50)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:78)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:164)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:478)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:520)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:22 7)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:94 1)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:409)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186 )

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:875 )

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandler Collection.java:250)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.jav a:149)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)

at org.eclipse.jetty.server.Server.handle(Server.java:349)

at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:441)

at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConne ction.java:919)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:582)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:51 )

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.jav a:586)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java :44)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598 )

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)

at java.lang.Thread.run(Thread.java:745)

Hazelcast config file:

openfire

    <password>openfire</password>

</group>

<management-center enabled="false"/>

<network>

    <port auto-increment="true">5701</port>
            <multicast-group>224.2.2.3</multicast-group>

            <multicast-port>54327</multicast-port>

X.Y…3.1:5701

X.Y.3.2:5701

Can anybody pls help here.

Best Regards,

SM

Hi Shil -

I am not familiar enough with the Rackspace cloud platform to know for sure, but it’s likely that the routing between your cluster members uses the WAN (public) network, rather than the LAN (private/local) network. Perhaps the hosts have multiple interfaces, which could allow private (LAN-based) routing between your cluster members? You might ask your tech rep at Rackspace to see if that is an option with their service.

However, per the Hazelcast docs, I believe you will need to secure an enterprise HZ license to enable WAN-based replication within your cluster. In that case, you may also be able to receive additional support building an appropriate Hazelcast configuration for the plugin, including an appropriate license key. The clustered Openfire deployments I have done to this point have been on private/local networks, and so do not typically encounter the timeout issues you are observing.

Regards,

Tom

Hi Tom,

Thank you very much for your response.

Yes, my Rackspace servers have multiple network interfaces, one of which is LAN interface, we are using those local IPs in the hazelcast config file:

X.Y…3.1:5701

X.Y.3.2:5701

Same for the DB, we are using the local LAN interface IP. We have also tried the same by stopping iptables on the two chat servers.

When this problem occurs, I did a netstat -anp | grep 5701, it shows a connection between X.Y.3,1 & X.Y.3.2
Also this problem is intermittent and only happens for one of the servers Chat02, meaning sometime the clustering page shows both the servers, some time it does not. The other server Chat01 can show both the servers Chat01 & Chat02.

Any insight would help.

Best Regards,

Shil

Same here… the same issue appears albeit we use private LAN among the 2 openfire servers we operate.

1 Like