Thread explosion in Smack

dabaohe · January 16, 2006, 3:02am

I program a C-S system. Jiver messager is used in server side and a app server(in Tomcat) is realized by Smack; the client is browser or handset.

now the system is runing. when the system run about 24 hours On a Debian Linux system, I found Tomcat(the web server) have more than 3000 threads. So I use jstack(java.sun.com) to see what happened in my system. the statistics data is as follow:

Thread information :number(thread count)

org.jivesoftware.smack.PacketReader.access$100(org.jivesoftware.smack.PacketRead er) @bci=1, line=43 (Interpreted frame) : 2700

java.net.SocketInputStream.socketRead0 : 92

org.jivesoftware.smack.PacketWriter.nextPacket() @bci=31, line=226 (Compiled frame) : 88

org.jivesoftware.smack.PacketWriter$KeepAliveTask.run() @bci=61, line=472 (Compiled frame) : 88

org.jivesoftware.smack.PacketWriter.processListeners() @bci=38, line=284 (Compiled frame) : 88

org.jivesoftware.smack.PacketReader.processListeners() @bci=160, line=257 (Interpreted frame) : 6

sun.jvm.hotspot.debugger.DebuggerException : 6

Total times: : 3068

The threads is accumulated by time until no response.

By the way, the total online users of my system is about 100 now, it is just a trial system.

the detail information about the first line(org.jivesoftware.smack.PacketReader.access$100(org.jivesoftware.smack.Pack etReader) @bci=1, line=43 (Interpreted frame) ) from jstack is like this:

Thread 9189: (state = BLOCKED)

java.lang.Object.wait(long) (Compiled frame; information may be imprecise)
java.lang.Object.wait() @bci=2, line=474 (Compiled frame)
org.jivesoftware.smack.PacketReader.access$100(org.jivesoftware.smack.PacketRea der) @bci=1, line=43 (Interpreted frame)
org.jivesoftware.smack.PacketReader$2.run() @bci=4, line=72 (Interpreted frame)

I read the source code of Smack in Packetwriter about 72th line and I really can’'t understand the code.

Can anyone tell me, why so man thread hang this, many many thanks.

LG1 · January 16, 2006, 6:56pm

Hi Dabao,

which version of Jive Messenger / Wildfire and Smack are you using?

Do you have a setup like this:?

Web-Client — Tomcat+Smack — JiveMessenger/Wildfire

Wildfire is not running in Tomcat, thus a stacktrace does not contain Wildfire classes.

It seems that every minute two threads are created, that’'s very unusual. I think the main question should be: Who does create these threads?

Imho every time you create an new XMPPConnection one PacketReader thread is created, maybe you never close the XMPP connections?

LG

dabaohe · January 17, 2006, 3:44am

Thank you for your help, My system’'s structure is as you say. I am consider for some reason about connection close.

dabaohe · January 17, 2006, 10:24am

The final issue:

In PacketReader, two thread are created:readerThread and listenerThread.

When XMPPConnection close() is invoke, the shutdown() in PacketReader is invoke, and done is set to ture, readerThread is end correctly. however listenerThread is not end normally and waiting in

private void processListeners() {

…

synchronized (listenerThread) {

listenerThread.wait();

}

…

}

Because listenerThread is a daemon Thread, so when main thread end, then the listenerThread is also end. But if this situation occured on a Web Service such as Tomcat application, the main thread is not end at all, the daemon thead is still alive and waiting now.

here is my test code:

/in PacketReader’‘s shutdown(), I add two System.out.println() to show two thread’'s status. you can try the code and input close for more times and see the output./

import org.jivesoftware.smack.XMPPConnection;

import java.io.BufferedReader;

import java.io.InputStreamReader;

public class Test extends Thread {

XMPPConnection con;

public void run() {

BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));

try {

System.out.print(“cmd>”);

String cmd = reader.readLine();

while(cmd != null && !cmd.equalsIgnoreCase(“exit”)) {

if(cmd.equalsIgnoreCase(“close”)) {

close();

}

System.out.print(“cmd>”);

cmd = reader.readLine();

}

}catch(Exception ex) {

ex.printStackTrace();

}

public void open() {

try {

con = new XMPPConnection(“192.168.1.107”);

}catch(Exception ex) {

ex.printStackTrace();

}

public void close() {

if(con != null) {

con.close();

}

/**

@param args

*/

public static void main(String[] args) {

Test t = new Test();

t.open();

t.start();

}

finally, I append a line code in PacketReader’'s shutdown() to release the object lock for listenerThread

…

done = true;

synchronized(listenerThread) {

listenerThread.notifyAll();

}

and the listenerThread now can close normally.

dabaohe · January 17, 2006, 10:30am

Thread is easy to use but difficult to debug.

LG1 · January 17, 2006, 11:12pm

Hi,

you could use and to make your code more readable.

Looking at Smack I wonder if this is a Smack bug in shutdown() in PacketReader.java.

It is not a servlet problem, you can also use a very simple code like this to get two Threads:

con = new XMPPConnection(Servername);

con.close();

con = new XMPPConnection(Servername);

con.close();

System.out.println("");

/code

connectionListeners is empty and will alway be empty except someone calls addConnectionListener() and this should only be the debugger.

Afaik Spark terminates and restarts itself after losing an XMPP connection, so a normal client will never experience this problem.

protected List connectionListeners = new ArrayList();

public void shutdown() {

if (!done) {

ArrayList listenersCopy;

synchronized (connectionListeners) {

listenersCopy = new ArrayList(connectionListeners); // always empty

for (Iterator i=listenersCopy.iterator(); i.hasNext(); ) {

ConnectionListener listener = (ConnectionListener)i.next();

listener.connectionClosed();

}

done = true;

}

/code

LG

dabaohe · January 18, 2006, 3:32am

test for and /code

//this is a test.

int i = 0;

for(int i = 0; i < 100; i++) {

System.out.println(i);

}

/code

dabaohe · January 18, 2006, 3:45am

The key problem is not connectionListeners, but listenerThread in PacketReader.

when PacketReader is created, readerThread and listenerThread are also created and started.

When XMPPConnection.close() is invoked, PacketReader.shutdown() is also invoked. The key point in shutdown is after

done = ture;

/code

readerThread and listenerThread should also be closed. For test if the two Thread are closed. I rewrite the shutdown() of

PacketReader:

public void shutdown() {

if (!done) {

ArrayList listenersCopy;

synchronized (connectionListeners) {

// Make a copy since it’'s possible that a listener will be

// removed from the list

listenersCopy = new ArrayList(connectionListeners);

for (Iterator i = listenersCopy.iterator(); i.hasNext() {

ConnectionListener listener = (ConnectionListener) i.next();

listener.connectionClosed();

}

done = true;//after this listenerThread and readerThread should be closed.

//this is my code for close listenerThread

//synchronized (listenerThread) {

// listenerThread.notifyAll();

//}

//Thread t to show readerThread and listenerThread’'s status

Thread t = new Thread() {

public void run() {

while (readerThread.isAlive() || listenerThread.isAlive()) {

System.out.println(“readerThread is alive:”

readerThread.isAlive());

System.out.println(“listenerThread is alive:”

listenerThread.isAlive());

try {

sleep(500);

} catch (Exception ex) {

ex.printStackTrace();

}

};

t.start();

}

/code

this is my test program:

import org.jivesoftware.smack.XMPPConnection;

import java.io.BufferedReader;

import java.io.InputStreamReader;

public class Test extends Thread {

XMPPConnection con;

public void run() {

BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));

try {

System.out.print(“cmd>”);

String cmd = reader.readLine();

while(cmd != null && !cmd.equalsIgnoreCase(“exit”)) {

if(cmd.equalsIgnoreCase(“close”)) {

close();

}

System.out.print(“cmd>”);

cmd = reader.readLine();

}

}catch(Exception ex) {

ex.printStackTrace();

}

public void open() {

try {

con = new XMPPConnection(“192.168.1.107”);

}catch(Exception ex) {

ex.printStackTrace();

}

public void close() {

if(con != null) {

con.close();

}

/**

@param args

*/

public static void main(String[] args) {

Test t = new Test();

t.open();

t.start();

}

/code

In my test program, XMPPConnection is opened in Test.open(), and you can input a close to close it. however, when you closed

the XMPPConnection, The Test is still running. Now you can see the output of Thread t in PacketReader, it show that

listenerThread is still alive.

However, when

synchronized (listenerThread) {

listenerThread.notifyAll();

}

/code

is added to shutdown() in PacketReader, When XMPPConnection.close() is invoked and PacketReader.shutdown() is invoked too,

readerThread and listenerThread can close normally, espacially for listenerThread.

Above all, the key problem is listenerThread cann’'t close normally because it is blocked in wait(). Because listenerThread in

PacketReader is a daemon Thread, so if your application is single Application, When main Thread end, the listenerThread will

be closed, but if you closed XMPPConnection but your main Thread still running, you can see PacketReader.listenerThread is

still running, precisely say it is blocking.

My English is poor, pardon me.

LG1 · January 18, 2006, 9:14pm

Hi Dabao,

is there a reason why you use notifyAll() instead of notify() ?

LG

PacketReader.java

/**

Shuts the packet reader down.

*/

public void shutdown() {

// Notify connection listeners of the connection closing if done hasn’'t already been set.

if (!done) {

ArrayList listenersCopy;

synchronized (connectionListeners) {

// Make a copy since it’'s possible that a listener will be removed from the list

listenersCopy = new ArrayList(connectionListeners);

for (Iterator i=listenersCopy.iterator(); i.hasNext(); ) {

ConnectionListener listener = (ConnectionListener)i.next();

listener.connectionClosed();

}

done = true;

// dear developer, please add these four lines and think about notify vs notifyAll:

synchronized (listenerThread)

{

listenerThread.notify(); // or notifyAll(); ?

};

}

/code

dabaohe · January 19, 2006, 1:48am

I really really don’'t think about it. This is an advice? can you explain it to me?

LG1 · January 26, 2006, 5:06pm

Hi Alex / Gato,

could you create an issue for this so it gets fixed? Or is it already fixed (I didn’'t look at the 1.5 code)?

LG

Alex_Wenckus · January 26, 2006, 6:32pm

Hey guys:

Thanks for the detailed bug report guys, the fix will be available in 2.2.0. SMACK-120

Alex

Fix · February 23, 2006, 5:17pm

Is there a workaround? Or do i have to wait for the next release?

When is the next release expected?

Thanks…

LG1 · February 23, 2006, 7:33pm

Hi,

the fix is included in the nightly builds which are available here: http://www.jivesoftware.org/nightly.jsp

Or you may just fix it for yourself as described above to keep the “current” Smack version.

LG

Fix · February 24, 2006, 10:05am

Thanks for the solution, it works better now.