HazelCast PluginServlet RoomRemovedEvent ClassCastException

OF v4.6.3
RestAPI Plugin v 1.4.0
HazelCast Plugin v2.5.0

These errors are common on our system

2021.04.26 14:04:09 org.jivesoftware.openfire.container.PluginServlet - Failed to serialize 'org.jivesoftware.openfire.plugin.util.cache.ClusteredCacheFactory$CallableTask'
com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'org.jivesoftware.openfire.plugin.util.cache.ClusteredCacheFactory$CallableTask'
	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:82) ~[?:?]
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:157) ~[?:?]
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:133) ~[?:?]
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:118) ~[?:?]
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:106) ~[?:?]
	at com.hazelcast.spi.impl.NodeEngineImpl.toData(NodeEngineImpl.java:319) ~[?:?]
	at com.hazelcast.executor.impl.ExecutorServiceProxy.submitToMembers(ExecutorServiceProxy.java:336) ~[?:?]
	at org.jivesoftware.openfire.plugin.util.cache.ClusteredCacheFactory.doClusterTask(ClusteredCacheFactory.java:372) ~[?:?]
	at org.jivesoftware.util.cache.CacheFactory.doClusterTask(CacheFactory.java:695) ~[xmppserver-4.6.3.jar:4.6.3]
	at org.jivesoftware.openfire.plugin.rest.controller.MUCRoomController.createRoom(MUCRoomController.java:290) ~[?:?]
	at org.jivesoftware.openfire.plugin.rest.controller.MUCRoomController.createChatRoom(MUCRoomController.java:160) ~[?:?]
	at org.jivesoftware.openfire.plugin.rest.service.MUCRoomService.createMUCRoom(MUCRoomService.java:53) ~[?:?]
	at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[?:?]
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[?:?]
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[?:?]
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[?:?]
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[?:?]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[?:?]
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[?:?]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[?:?]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[?:?]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[?:?]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[?:?]
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[?:?]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[?:?]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[?:?]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
	at org.jivesoftware.openfire.container.PluginServlet.handleServlet(PluginServlet.java:461) ~[xmppserver-4.6.3.jar:4.6.3]
	at org.jivesoftware.openfire.container.PluginServlet.service(PluginServlet.java:134) ~[xmppserver-4.6.3.jar:4.6.3]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:226) ~[xmppserver-4.6.3.jar:4.6.3]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:234) ~[xmppserver-4.6.3.jar:4.6.3]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129) ~[sitemesh-2.4.2.jar:?]
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77) ~[sitemesh-2.4.2.jar:?]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[jetty-security-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) [jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.jivesoftware.openfire.muc.MUCRole$Role
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_282]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_282]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_282]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_282]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_282]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_282]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_282]
	at org.jivesoftware.openfire.muc.spi.LocalMUCRoom.writeExternal(LocalMUCRoom.java:3286) ~[xmppserver-4.6.3.jar:4.6.3]
	at org.jivesoftware.openfire.muc.cluster.RoomUpdatedEvent.writeExternal(RoomUpdatedEvent.java:62) ~[xmppserver-4.6.3.jar:4.6.3]
	at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1459) ~[?:1.8.0_282]
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1430) ~[?:1.8.0_282]
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) ~[?:1.8.0_282]
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) ~[?:1.8.0_282]
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) ~[?:1.8.0_282]
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) ~[?:1.8.0_282]
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) ~[?:1.8.0_282]
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[?:1.8.0_282]
	at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.write(JavaDefaultSerializers.java:116) ~[?:?]
	at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.write(JavaDefaultSerializers.java:109) ~[?:?]
	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43) ~[?:?]
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:154) ~[?:?]
	... 81 more

It looks like this exception is thrown when we use the RestAPI to delete MUCs that existed in our OF v4.5.2 installation prior to upgrade to v4.6.3. With a MUC that was created under v4.6.3, we don’t see this exception. This implies we need to convert old MUCs? If yes, then how do we convert them?

A known source of ClassCastExceptions is an issue that occurs when a plugin that makes use of a clustered cache gets reloaded/restarted/updated. This should only affect instances that also make use of the Hazelcast plugin (which, by the looks of the stacktrace that was provided, you are). If that’s the case, you might be able to work around the issue by restarting Openfire.

@guus thank you for the reply. Yes we’re using Hazelcast plugin v2.5.0

Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.jivesoftware.openfire.muc.MUCRole$Role

Seems to be key.

We restarted after installing the update and subsequently afterwards. I expect to be able to reproduce this issue by installing 4.5.2, creating a MUC, then upgrading to v4.6.3. I’ll verify this expected result and include restarting after upgrade to 4.6.3.

Ah, that is a different problem. What’s the stacktrace for that ‘string cannot be cast as Role’ exception?

This might be caused by running different versions of Openfire in the same cluster, but I’m not sure.

Actually, it looks like we’re seeing this error even with MUCs that were created in v4.6.3 also, but not always – it seems to depend on something unknown. It seems more common with MUCs created in v4.5.2.

Server restarts have no effect on this. From what I’ve seen, I’ll be surprised if this only affects us.

For background: that’s the full stacktrace from the error.log. JVM is truncating the stacktrace, according to https://docs.oracle.com/javase/6/docs/api/java/lang/Throwable.html#printStackTrace(). The first occurrence of the exception should be non-truncated.

I’m not sure if this is the first occurrence with the full stack trace:

2021.04.26 19:09:42 org.jivesoftware.openfire.container.PluginServlet - org.jivesoftware.openfire.muc.MUCRole$Role cannot be cast to java.lang.String (through reference chain: org.jivesoftware.openfire.plugin.rest.entity.MUCRoomEntity["broadcastPresenceRoles"]->java.util.UnmodifiableRandomAccessList[0])
org.codehaus.jackson.map.JsonMappingException: org.jivesoftware.openfire.muc.MUCRole$Role cannot be cast to java.lang.String (through reference chain: org.jivesoftware.openfire.plugin.rest.entity.MUCRoomEntity["broadcastPresenceRoles"]->java.util.UnmodifiableRandomAccessList[0])
        at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:218) ~[?:?]
        at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:197) ~[?:?]
        at org.codehaus.jackson.map.ser.std.SerializerBase.wrapAndThrow(SerializerBase.java:166) ~[?:?]
        at org.codehaus.jackson.map.ser.std.IndexedStringListSerializer.serializeContents(IndexedStringListSerializer.java:99) ~[?:?]
        at org.codehaus.jackson.map.ser.std.IndexedStringListSerializer.serialize(IndexedStringListSerializer.java:63) ~[?:?]
        at org.codehaus.jackson.map.ser.std.IndexedStringListSerializer.serialize(IndexedStringListSerializer.java:24) ~[?:?]
        at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446) ~[?:?]
        at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150) ~[?:?]
        at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112) ~[?:?]
        at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:610) ~[?:?]
        at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256) ~[?:?]
        at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1613) ~[?:?]
        at org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:559) ~[?:?]
        at com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy.writeTo(JacksonProviderProxy.java:160) ~[?:?]
        at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302) ~[?:?]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510) ~[?:?]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[?:?]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[?:?]
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[?:?]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[?:?]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[?:?]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
        at org.jivesoftware.openfire.container.PluginServlet.handleServlet(PluginServlet.java:461) ~[xmppserver-4.6.3.jar:4.6.3]
        at org.jivesoftware.openfire.container.PluginServlet.service(PluginServlet.java:134) ~[xmppserver-4.6.3.jar:4.6.3]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:226) ~[xmppserver-4.6.3.jar:4.6.3]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:234) ~[xmppserver-4.6.3.jar:4.6.3]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129) ~[sitemesh-2.4.2.jar:?]
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77) ~[sitemesh-2.4.2.jar:?]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[jetty-security-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) [jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905) [jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.lang.ClassCastException: org.jivesoftware.openfire.muc.MUCRole$Role cannot be cast to java.lang.String
        at org.codehaus.jackson.map.ser.std.IndexedStringListSerializer.serializeContents(IndexedStringListSerializer.java:91) ~[?:?]
        ... 71 more

It’s the same error with RestAPI plugin v1.4.0 getAllRooms endpoint. Also FWIW all of our nodes are confirmed on the same OF version v4.6.3.

@guus it’s also worth noting that MUCs created directly from v4.6.3 admin have no such problems and seem to be functioning nominally across the cluster and from our XMPP client.

I submitted a PR to address this issue at https://github.com/igniterealtime/Openfire/pull/1831. This issue also happens on create and update.

RestAPI v1.4.0 is incompatible with OF v4.6.x. v1.5.0 is compatible (almost) but must be built from source presently.

This is related to Error 500 retrieving rooms info with REST API plugin

This indeed seems to be the same issue. This problem will be resolved in the upcoming version 1.6.0 of the REST API plugin. A snapshot/prerelease download can be obtained from Ignite Realtime: Openfire Plugins