Just wanted to confirm before we report this as a bug. The latest version of Openfire is not compatible with the Hazelcast plugin correct?
We are seeing an exception on the server and all access to the administration console is lost.
Can someone confirm that this is expected at this point in time and if possible provide a time frame for an update of the plugin.
Thanks
This is incorrect. The Hazelcast plugin (version 2.6.1) is compatible with Openfire 4.8.0.
What exceptions are you seeing? Do you seen the same exceptions when you’re not using the Hazelcast plugin?
The exception we are seeing is specific to the hazelcast plugin and we are seeing that pade is also crashing. We are not seeing any other error in the log.
Let me go through the logs and i will post the exact messages,
Hi @guus so the exception we are seeing is the following:
2024.03.19 11:46:37 WARN [hz.openfire.event-4]: com.hazelcast.spi.EventService - [192.168.131.80]:5701 [openfire] [3.12.5] Error while logging processing event
com.hazelcast.nio.serialization.HazelcastSerializationException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:70) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:193) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.DataAwareEntryEvent.getValue(DataAwareEntryEvent.java:90) ~[hazelcast-3.12.5.jar:3.12.5]
at org.jivesoftware.openfire.plugin.util.cache.ClusteredCache$1.entryUpdated(ClusteredCache.java:110) ~[hazelcast-2.6.1.jar:?]
at com.hazelcast.map.impl.MapListenerAdaptors$4$1.onEvent(MapListenerAdaptors.java:129) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.MapListenerAdaptors$4$1.onEvent(MapListenerAdaptors.java:126) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.event.MapEventPublishingService.callListener(MapEventPublishingService.java:172) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:184) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:83) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:47) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:105) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:78) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.eventservice.impl.EventProcessor.process(EventProcessor.java:61) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.eventservice.impl.RemoteEventProcessor.run(RemoteEventProcessor.java:47) [hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:244) [hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:227) [hazelcast-3.12.5.jar:3.12.5]
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:184) ~[hazelcast-3.12.5.jar:3.12.5]
As discussed we are also losing all access to the administration console. But there is nothing logged. I will try to recreate this and post back with any interesting findings.
Ok so I took another turn on this. And when we upgrade to 4.8.1 from 4.7.5 we see this error popping up every 5 seconds
2024.03.20 09:48:52 ERROR [socket_c2s-thread-2]: org.jivesoftware.openfire.muc.spi.MultiUserChatServiceImpl - Internal server error
and then on the other node:
2024.03.20 09:57:38 ERROR [socket_c2s-thread-5]: org.jivesoftware.openfire.handler.PresenceUpdateHandler - Internal server error. Triggered by packet:
2024.03.20 09:57:39 ERROR [socket_c2s-thread-3]: org.jivesoftware.openfire.IQRouter - Could not route packet
Also for the admin console we are getting the following errors
HTTP ERROR 500 java.lang.NullPointerException: Cannot invoke "org.jivesoftware.openfire.user.User.getUsername()" because the return value of "org.jivesoftware.util.WebManager.getUser()" is null
URI: /plugin-admin.jsp
STATUS: 500
MESSAGE: java.lang.NullPointerException: Cannot invoke "org.jivesoftware.openfire.user.User.getUsername()" because the return value of "org.jivesoftware.util.WebManager.getUser()" is null
SERVLET: org.jivesoftware.openfire.admin.plugin_002dadmin_jsp
CAUSED BY: java.lang.NullPointerException: Cannot invoke "org.jivesoftware.openfire.user.User.getUsername()" because the return value of "org.jivesoftware.util.WebManager.getUser()" is null
Caused by:
java.lang.NullPointerException: Cannot invoke "org.jivesoftware.openfire.user.User.getUsername()" because the return value of "org.jivesoftware.util.WebManager.getUser()" is null
at org.jivesoftware.util.WebManager.logEvent(WebManager.java:153)
at org.jivesoftware.openfire.admin.plugin_002dadmin_jsp._jspService(plugin_002dadmin_jsp.java:292)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.jivesoftware.util.LocaleFilter.doFilter(LocaleFilter.java:73)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.jivesoftware.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:49)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:174)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:292)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.jivesoftware.admin.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:53)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
at java.base/java.lang.Thread.run(Thread.java:833)
It might help to notice that we are running the cluster on hybrid auth.
Unfortunately this is in production and there is not much more i can do about this.
I will be reverting back to 4.7.5 until this is resolved.
Are the logs we have shared of any help in detecting the issue?
Are we the only ones facing issues with this version?
Could someone confirm that they are running a 4.8.1 cluster with hybrid auth without issues?
I thank you in advance.
It has been a few weeks since we started this thread but we have not received any feedback.
Today we had another go at transitioning our production environment to version 4.8.1 and again this failed. We noticed two issues. firstly the updated pade plugin (v1.8.1) does not load and on the plugin page there is a message relating to hazelcast exception.
Secondly we noticed that after a few minutes (maybe 10 minutes) we lose the ability to connect to the admin ui. We are not getting any error messages we just submit the login form the page reloads and nothing happens. Same behavior for https and http connections. Nothing is logged on the server nothing on the console of the browser.
I would like to reiterate my last question Is there anyone that has actually managed to make this work? We are running a clustered environment with hybrid auth and the pade plugin.
Any insight would be greatly appreciated. @guus @Dele_Olajide anyone??
I thank you in advance.
I have not done any testing for Pade 1.8.1 with Hazelcast and OF 4.8.1. Thank you for highlighting the issue. Not sure when I can look at this.
My advice for now is to stay on 4.7.5 if you can use Pade with Hazelcast in clustering mode.
Exactly what I was looking for.
4.7.5 runs flawlessly so I will wait until we have some more experience accumulated there.
Thanks @Dele_Olajide
Thanks for the logs. That will help debug the issue 