Websockets issue 3.x / 4.0 beta

Hello,

I’ve been trying to find the source of this issue for a while, websocket implementation hangs freezing new connections and pretty much everything but the admin layer.

I thought issue was due websockets plugin not being fully compliant, but after the v1.1 it shouldn’t be a problem, but it still is, issue is the same on both 3.x and 4.0, websocket 1.0 and 1.1.

I did some debugging with jconsole, no deadlocks, but I see a lot of Jetty-QTP-BOSH threads, they pile up until reach 250 - 300 thread limit, when this happens everything stops responding. Their stack is pretty much the same:

Name: Jetty-QTP-BOSH-188
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a230cbe
Total blocked: 0  Total waited: 2 Stack trace: sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
org.jivesoftware.openfire.websocket.XmppWebSocket.onTextMethod(XmppWebSocket.java:108)
sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:497)
org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:70)
org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72)
org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextMessage(JettyAnnotatedEventDriver.java:225)
org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:64)
org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextFrame(JettyAnnotatedEventDriver.java:217)
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:160)
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:163)
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:92)
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:123)
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:78)
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:628)
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:476)
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
java.lang.Thread.run(Thread.java:745)

From jconsole, you can see on the graph that the number of threads go up to around 320, that’s when it stop responding, I’ve killed java and up it goes again:

We’re running this using strophe as client, ws is accessed throught a nginx proxy.

ulimits, firewalls, etc all been checked, any clue of what’s going on? anyone ever got the same issue?

Thanks.

Hi Rodrigo -

Thanks for this report. By the looks of it, it’s possible the plugin’s local XMPPPacketReader pool is being exhausted and blocking the BOSH C2S threads. I will take a closer look to see if I can recreate this condition under test. We might need to provide some additional configuration options with the plugin to allow this resource pool to grow/shrink properly under load.

I have opened OF-1006 to track this issue.

Regards,

Tom

This solved the issue, server been up and running for over 15 days now.

Thanks