Error in cluster with Hazelcast. Hazelcast throw exception

Hello.
I’ have a cluster with 7 server in openfire 4.2.2 using hazelcast 2.2.4, today in the morning 4 servers with this error message:

Exception:

com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93)
at com.hazelcast.map.impl.proxy.MapProxySupport.toData(MapProxySupport.java:1122)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:81)
at org.jivesoftware.openfire.plugin.util.cache.ClusteredCache.get(ClusteredCache.java:92)
at org.jivesoftware.util.cache.CacheWrapper.get(CacheWrapper.java:136)
at org.jivesoftware.openfire.lockout.LockOutManager.getUserLockOut(LockOutManager.java:241)
at org.jivesoftware.openfire.lockout.LockOutManager.getDisabledStatus(LockOutManager.java:149)
at org.jivesoftware.openfire.lockout.LockOutManager.isAccountDisabled(LockOutManager.java:160)
at org.jivesoftware.openfire.auth.AuthFactory.authenticate(AuthFactory.java:199)
at org.jivesoftware.openfire.admin.login_jsp._jspService(login_jsp.java:189)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:39)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.jivesoftware.util.LocaleFilter.doFilter(LocaleFilter.java:73)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.jivesoftware.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:226)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:215)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)

In this 4 server the service openfire are running but the servers are out of the cluster.
The other 3 servers with some ghosts sessions

There’s a bug (https://issues.igniterealtime.org/browse/HZ-11) that means a server gets in an un-fixable state when it leaves the hazelcast cluster. This looks like an instance of that;

I can’t tell you why the servers left the cluster, though - perhaps some network problems overnight?

Greg