When we are upgrading from 4.6.4 version to 4.9.1 version .
openfire starts using lot of jvm memory and node cpu and finally crashing or throwing lot of websocket error in logs and many functionality stop working
with in 15 mins we had to revert back to old version again and unable to upgrade the openfire.
we are using below specificied vm configuration
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 48 bits physical, 48 bits virtual
CPU(s): 4
at a time we have approx 2000 users anytime. and also lot of restapi call for other adhoc functionality
This was our second attempt to upgrade openfire , but unfortunately couldn’t find success,
Could someone please help on this?
i have uploaded couple of grafana metrics usage also where memory cpu and network socket usage increase after upgrade to limit and again came down to normal when we reverted back to old openfire.
please let us know if you need info in debug further.
error in logs:
2024.11.26 03:37:56.573 e[33mWARN e[m [TaskEngine-pool-31]: org.jivesoftware.openfire.websocket.WebSocketClientConnectionHandler - Unable to send websocket ping to remote peer: WebSocketConnection{jid=test@hello.com/iphonexxx, remotePeer=/x.x.x.x:63080, socket=org.jivesoftware.openfire.websocket.WebSocketClientConnectionHandler@6e22047c, connectionType=SOCKET_C2S}
java.io.IOException: org.eclipse.jetty.io.EofException
at org.eclipse.jetty.util.FutureCallback.block(FutureCallback.java:163) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.FutureCallback.block(FutureCallback.java:139) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.common.JettyWebSocketRemoteEndpoint.sendBlocking(JettyWebSocketRemoteEndpoint.java:191) ~[websocket-jetty-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.common.JettyWebSocketRemoteEndpoint.sendPing(JettyWebSocketRemoteEndpoint.java:136) ~[websocket-jetty-common-10.0.18.jar:10.0.18]
at org.jivesoftware.openfire.websocket.WebSocketClientConnectionHandler$WebsocketFramePingTask.run(WebSocketClientConnectionHandler.java:264) [xmppserver-4.9.1.jar:4.9.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:116) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.ssl.SslConnection.networkFlush(SslConnection.java:486) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1115) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.FrameFlusher.process(FrameFlusher.java:338) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.enqueueFrame(WebSocketConnection.java:649) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$OutgoingAdaptor.sendFrame(WebSocketCoreSession.java:724) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.ExtensionStack.sendFrame(ExtensionStack.java:258) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$Flusher.forwardFrame(WebSocketCoreSession.java:815) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.FragmentingFlusher.onFrame(FragmentingFlusher.java:50) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.TransformingFlusher$Flusher.process(TransformingFlusher.java:163) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.sendFrame(TransformingFlusher.java:77) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.sendFrame(WebSocketCoreSession.java:526) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.common.JettyWebSocketRemoteEndpoint.sendBlocking(JettyWebSocketRemoteEndpoint.java:190) ~[websocket-jetty-common-10.0.18.jar:10.0.18]
... 7 more
Suppressed: org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:116) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.ssl.SslConnection.networkFlush(SslConnection.java:486) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1115) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doShutdownOutput(SslConnection.java:1338) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doClose(SslConnection.java:1434) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:258) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:227) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1204) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.FrameFlusher.process(FrameFlusher.java:338) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.enqueueFrame(WebSocketConnection.java:649) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$OutgoingAdaptor.sendFrame(WebSocketCoreSession.java:724) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.ExtensionStack.sendFrame(ExtensionStack.java:258) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$Flusher.forwardFrame(WebSocketCoreSession.java:815) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.FragmentingFlusher.onFrame(FragmentingFlusher.java:50) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.TransformingFlusher$Flusher.process(TransformingFlusher.java:163) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.sendFrame(TransformingFlusher.java:77) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.sendFrame(WebSocketCoreSession.java:526) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.common.JettyWebSocketRemoteEndpoint.sendBlocking(JettyWebSocketRemoteEndpoint.java:190) ~[websocket-jetty-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.common.JettyWebSocketRemoteEndpoint.sendPing(JettyWebSocketRemoteEndpoint.java:136) ~[websocket-jetty-common-10.0.18.jar:10.0.18]
at org.jivesoftware.openfire.websocket.WebSocketClientConnectionHandler$WebsocketFramePingTask.run(WebSocketClientConnectionHandler.java:264) [xmppserver-4.9.1.jar:4.9.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[?:?]
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[?:?]
at sun.nio.ch.IOUtil.write(IOUtil.java:182) ~[?:?]
at sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[?:?]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493) ~[?:?]
at java.nio.channels.SocketChannel.write(SocketChannel.java:507) ~[?:?]
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:110) ~[jetty-io-10.0.18.jar:10.0.18]
... 32 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[?:?]
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[?:?]
at sun.nio.ch.IOUtil.write(IOUtil.java:182) ~[?:?]
at sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[?:?]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493) ~[?:?]
at java.nio.channels.SocketChannel.write(SocketChannel.java:507) ~[?:?]
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:110) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.ssl.SslConnection.networkFlush(SslConnection.java:486) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1115) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386) ~[jetty-io-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.FrameFlusher.process(FrameFlusher.java:338) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.enqueueFrame(WebSocketConnection.java:649) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$OutgoingAdaptor.sendFrame(WebSocketCoreSession.java:724) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.ExtensionStack.sendFrame(ExtensionStack.java:258) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$Flusher.forwardFrame(WebSocketCoreSession.java:815) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.FragmentingFlusher.onFrame(FragmentingFlusher.java:50) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.TransformingFlusher$Flusher.process(TransformingFlusher.java:163) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.sendFrame(TransformingFlusher.java:77) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.sendFrame(WebSocketCoreSession.java:526) ~[websocket-core-common-10.0.18.jar:10.0.18]
at org.eclipse.jetty.websocket.common.JettyWebSocketRemoteEndpoint.sendBlocking(JettyWebSocketRemoteEndpoint.java:190) ~[websocket-jetty-common-10.0.18.jar:10.0.18]
