We have Openfire 3.5.2 running with currently approx. 200 concurrent connections with most of them being SSL connections. After a few hours the memory and CPU consumption rises from 20 MB / 1% to 350 MB / 10%. After some days it sometimes even jumps to about 900MB. At this point connections become unreliable and we have to restart the server.
ps -T -p 5779 -o pid,tid,pri,time | grep -v ‘00:00:00’ shows 1300 threads.
The most active thread is
5779 5790 24 00:23:02 XX
In nohup.out this is the “VM Thread”.
“VM Thread” prio=10 tid=0x08119400 nid=0x169e runnable
“GC task thread#0 (ParallelGC)” prio=10 tid=0x0805e400 nid=0x169a runnable
“GC task thread#1 (ParallelGC)” prio=10 tid=0x0805f400 nid=0x169b runnable
“VM Periodic Task Thread” prio=10 tid=0x08134400 nid=0x16a5 waiting on condition
JNI global references: 807
Heap
PSYoungGen total 169280K, used 110135K [0xe68f0000, 0xf3850000, 0xf4c70000)
eden space 134208K, 61% used [0xe68f0000,0xeb9e3d78,0xeec00000)
from space 35072K, 77% used [0xeec00000,0xf0699f00,0xf0e40000)
to space 34496K, 0% used [0xf16a0000,0xf16a0000,0xf3850000)
PSOldGen total 440576K, used 231752K [0x74c70000, 0x8fab0000, 0xe68f0000)
object space 440576K, 52% used [0x74c70000,0x82ec21f8,0x8fab0000)
PSPermGen total 30208K, used 29287K [0x70c70000, 0x729f0000, 0x74c70000)
object space 30208K, 96% used [0x70c70000,0x72909e40,0x729f0000)
We also see several hundred “Thread-n” threads.
“Timer-9801” daemon prio=10 tid=0x4e895800 nid=0x7db9 in Object.wait() http://0x32158000…0x321590b0
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Unknown Source)
- locked <0x7fc03a20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Unknown Source)
and a lot of “Smack Reconnection Manager” threads.
“Smack Reconnection Manager” daemon prio=10 tid=0x09674c00 nid=0x3fb3 waiting on condition http://0x2e2b2000…0x2e2b2e30
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x775b05e0> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unk nown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterrupti bly(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibl y(Unknown Source)
at java.util.concurrent.Semaphore.acquire(Unknown Source)
at org.jivesoftware.smack.PacketReader.startup(PacketReader.java:151)
at org.jivesoftware.smack.XMPPConnection.initConnection(XMPPConnection.java:1066)
at org.jivesoftware.smack.XMPPConnection.connectUsingConfiguration(XMPPConnection. java:1025)
at org.jivesoftware.smack.XMPPConnection.connect(XMPPConnection.java:1536)
at org.jivesoftware.smack.ReconnectionManager$2.run(ReconnectionManager.java:174)
The beginning of nohup.opt also shows a few hundred of the following exceptions:
java.io.EOFException: no more data available - expected end tag </stream:stream> to close start tag stream:stream from line 1, parser stopped on END_TAG seen …</stream:features> … @1:343
at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:3035)
at org.xmlpull.mxp1.MXParser.more(MXParser.java:3046)
at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1384)
at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
at org.jivesoftware.smack.PacketReader.parsePackets(PacketReader.java:368)
at org.jivesoftware.smack.PacketReader.access$000(PacketReader.java:44)
at org.jivesoftware.smack.PacketReader$1.run(PacketReader.java:76)
java.net.SocketException: Connection closed by remote host
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkWrite(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(Unknown Source)
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
at sun.nio.cs.StreamEncoder.flush(Unknown Source)
at java.io.OutputStreamWriter.flush(Unknown Source)
at java.io.BufferedWriter.flush(Unknown Source)
at org.jivesoftware.smack.PacketWriter.writePackets(PacketWriter.java:274)
at org.jivesoftware.smack.PacketWriter.access$000(PacketWriter.java:40)
java.io.EOFException: input contained no data
at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:3003)
at org.xmlpull.mxp1.MXParser.more(MXParser.java:3046)
at org.xmlpull.mxp1.MXParser.parseProlog(MXParser.java:1410)
at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1395)
at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
at org.jivesoftware.smack.PacketReader.parsePackets(PacketReader.java:368)
at org.jivesoftware.smack.PacketReader.access$000(PacketReader.java:44)
at org.jivesoftware.smack.PacketReader$1.run(PacketReader.java:76)
java.io.EOFException: no more data available - expected end tag </stream:stream> to close start tag stream:stream from line 1, parser stopped on END_TAG seen …</stream:features>… @1:342
at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:3035)
at org.xmlpull.mxp1.MXParser.more(MXParser.java:3046)
at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1144)
at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
at org.jivesoftware.smack.PacketReader.parsePackets(PacketReader.java:368)
at org.jivesoftware.smack.PacketReader.access$000(PacketReader.java:44)
at org.jivesoftware.smack.PacketReader$1.run(PacketReader.java:76)
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(Unknown Source)
at com.sun.net.ssl.internal.ssl.OutputRecord.write(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(Unknown Source)
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
at sun.nio.cs.StreamEncoder.flush(Unknown Source)
at java.io.OutputStreamWriter.flush(Unknown Source)
at java.io.BufferedWriter.flush(Unknown Source)
at org.jivesoftware.smack.PacketWriter.writePackets(PacketWriter.java:274)
at org.jivesoftware.smack.PacketWriter.access$000(PacketWriter.java:40)
at org.jivesoftware.smack.PacketWriter$1.run(PacketWriter.java:87)
The server is only using the Gateway 1.2.3 plugin.
Any idea what could be causing this? Is there a problem with the connection cleanup?
Any help would be appreciated.
Thanks,
Henrik