powered by Jive Software

Smack-4.4.2 No response from server error when connecting more than 300 clients

We are getting “No response received within reply timeout. Timeout was 5000ms (~5s). While waiting for successful SASL authentication” exception in all of our smack clients when we try to connect 300 clients at same time.

Server Machine Environment:
Operating System: Microsoft Windows 7 Professional
System Type: 64-bit Operating System
Processor: Intel(R) Core™2 Duo CPU E7500 @ 2.93GHz, 2 core(s)
RAM: 4GB

Openfire Environment:
Version: Openfire-4.0.4.
Java memory: We customized the minimum and maximum memory size to 600MB.

openfire-service.vmoptions
-Xms600m
-Xmx600m

Smack version: 4.4.2

We are using a Java based Windows desktop client built with smack version 4.4.2 to communicate with an Openfire server. We are running Openfire as a Windows Service.
We don’t face any issues when we try to connect the clients 50 by 50 up to 300. After connecting all 300 clients and establishing communication, we attempted a Openfire server restart.
The clients will be pinging the server continuously until it is up.
After the server is up, all of our clients will try to connect with the server. At the time all the clients are throwing the exception “No response received within reply timeout.”. No matter how many times it tries to reconnect the same exception is received.

Exception:

org.jivesoftware.smack.SmackException$NoResponseException: No response received within reply timeout. Timeout was 5000ms (~5s). While waiting for successful SASL authentication [XMPPTCPConnection[not-authenticated] (0)]
at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:99)
at org.jivesoftware.smack.sasl.SASLMechanism.throwExceptionIfRequired(SASLMechanism.java:339)
at org.jivesoftware.smack.SASLAuthentication.authenticate(SASLAuthentication.java:211)
at org.jivesoftware.smack.AbstractXMPPConnection.authenticate(AbstractXMPPConnection.java:904)
at org.jivesoftware.smack.tcp.XMPPTCPConnection.loginInternal(XMPPTCPConnection.java:381)
at org.jivesoftware.smack.AbstractXMPPConnection.login(AbstractXMPPConnection.java:641)
at handler.LoginHandler.login(LoginHandler.java:30)

We referred forums and tried increasing the reply timeout to 20 seconds in all clients(300) using the “setReplyTimeout(20000)” method of the XMPPTCPConnection class. But still the issue is not resolved in the Openfire server restart case mentioned above.
After we met with the issue, without stopping the Openfire server we modified a few of the clients reply timeout up to 80 seconds. Still the clients through the same exception.
Also after the Openfire restarted when clients try to connect, the Openfire Java memory is spiking to the maximum.

The only way to resolve the issue is to restart each client.
Please help us to find more on this and resolve the issue.

Thanks in advance!

That sounds more like an internal Smack issue, but I would need a thread dump and an XMPP trace to tell more. See also https://github.com/igniterealtime/Smack/wiki/How-to-ask-for-help,-report-an-issue-and-possible-solve-the-problem-yourself

You should also update to the latest released Smack version, 4.4.4. Maybe it fixes the issue for you.

Thanks for your immediate response. As of now we don’t have any thread dump backup of the issue and not sure which thread dump you are asking(either Openfire or client side).

Regarding XMPP trace in Openfire, we don’t see any stream messages and exception at the time of issue.

We didn’t enable the smack debug configuration at the time of our testing in clients. Will enable it and test once and update you the results as soon as possible.
It will be really good if you can help more on taking thread dump.

Since it is Smack which appears to “hang”, a thread dump of the application using Smack would be a good start. :slight_smile:

I did the testing again with same setup mentioned above to recreate the issue.
I have enabled the smack debugger and also increased the XMPPTCPConnection’s reply timeout to 20000ms(setReplyTimeout(20000)).
I recreated the issue successfully and got the XMPP trace details.

Smack version: 4.4.2

Logs from a client’s Smack debugger:

15:36:01 SENT (7): <stream:stream xmlns='jabber:client' to='10.10.1.45' xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xml:lang='en-US'>
15:36:25 SENT (8): <stream:stream xmlns='jabber:client' to='10.10.1.45' xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xml:lang='en-US'>
15:36:48 SENT (9): <stream:stream xmlns='jabber:client' to='10.10.1.45' xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xml:lang='en-US'>

This is the client’s request to Openfire server after one request is failed(We have separate mechanism in client side to retry continuously).
Each request will be sent after one is failed with “No response from server” Exception.

15:36:21,783 ERROR [connection.XmppConn] [84]: Exception while connecting to XMPP :  
org.jivesoftware.smack.SmackException$NoResponseException: No response received within reply timeout. Timeout was 20000ms (~20s). While waiting for establishing TLS [XMPPTCPConnection[not-authenticated] (7)]
	at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:99)
	at org.jivesoftware.smack.AbstractXMPPConnection.waitForConditionOrConnectionException(AbstractXMPPConnection.java:732)
	at org.jivesoftware.smack.AbstractXMPPConnection.waitForConditionOrThrowConnectionException(AbstractXMPPConnection.java:737)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectInternal(XMPPTCPConnection.java:854)
	at org.jivesoftware.smack.AbstractXMPPConnection.connect(AbstractXMPPConnection.java:529)
	at connection.XmppConn.connect(XmppConn.java:75)

Openfire’s “info.log” at the time of client request(These details are filtered by using client’s IP Address:15.207.84.54):

15:35:49 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - SENT - (   14231197): <message to="0b00a8e86b00ebf@10.10.1.45/Desktop" id="1MSGF-9" type="chat" from="emesstomcat@10.10.1.45/Smack"><body>&lt;Login&gt;
15:35:49 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - RECV - (   14231197): <message to='emesstomcat@10.10.1.45/Smack' id='UUSMX-4' type='chat'><body>&lt;History>
15:36:07 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - SENT - (   14231197): <a xmlns='urn:xmpp:sm:3' h='5' />
15:36:12 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - SENT - (   14231197): <message to="0b00a8e86b00ebf@10.10.1.45/Desktop" id="1MSGF-13" type="chat" from="emesstomcat@10.10.1.45/Smack"><body>&lt;HistoryResponse&gt;&lt;/HistoryResponse&gt;</body></message>
15:36:12 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - RECV - (   14231197): <iq to='10.10.1.45' id='UUSMX-5' type='get'><ping xmlns='urn:xmpp:ping'/></iq>
15:36:25 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - SENT - (   14231197): <iq type="result" id="UUSMX-5" from="10.10.1.45" to="0b00a8e86b00ebf@10.10.1.45/Desktop"/>
15:36:25 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - SENT - (   14231197): <r xmlns='urn:xmpp:sm:3' />
15:36:25 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - RECV - (   14231197): <a xmlns='urn:xmpp:sm:3' h='5'/>
15:36:25 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - SENT - (   14231197):  
15:36:48 org.jivesoftware.openfire.plugin.RawPrintFilter - SSL /15.207.84.54:49849 - RECV - (   14231197): <iq to='10.10.1.45' id='UUSMX-6' type='get'><ping xmlns='urn:xmpp:ping'/></iq>

I am attaching the smack debugger log file, Thread dump of the client and Openfire’s thread dump.
Client’s smack debugger log:
XMPP_Trace.log (121.8 KB)

Openfire’s Thread Dump:
Openfire_15.37_all.txt (185.4 KB)

Hope you can get what I am trying to convey. Please let me know if you need any more details.
Thanks in advance!!