Java out of memory

Openfire Openfire Support
1

Hi,

I am running with serios problem in openfire, please help me to tune it to work fine.

I am running Openfire 3.8.2 with ldap backend. I have around 8000 users in ldap database and in that avg 2000 users will be concorrent online users in openfire.

Initially I didnt use connection manager to run the server. once in a week or byweek server shows out of memory and I forced to restart the server to work again.Then i decided to use connection manager after going through blogs. After using connection manager, openfire is running good but connection manager running out of memory.

I am using jre1.7.0_45

memory setting of openfire: OPENFIRE_OPTS="-Xmx4596m"

memory setting of connection manager : JVM_SETTINGS="-Xms512m -Xmx3024m"

error i am getting in connection manager when its going out of memory

2014.03.01 08:58:39 [org.jivesoftware.multiplexer.net.ConnectionHandler.exceptionCaught(ConnectionH andler.java:107)]

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.(Unknown Source)

at java.nio.ByteBuffer.allocate(Unknown Source)

at org.apache.mina.filter.support.SSLByteBufferPool.createBuffer(SSLByteBufferPool .java:175)

at org.apache.mina.filter.support.SSLByteBufferPool.expandBuffer(SSLByteBufferPool .java:144)

at org.apache.mina.filter.support.SSLHandler.encrypt(SSLHandler.java:359)

at org.apache.mina.filter.SSLFilter.filterWrite(SSLFilter.java:479)

at org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(Ab stractIoFilterChain.java:361)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1300(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(Ab stractIoFilterChain.java:659)

at org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilte r.java:210)

at org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(Ab stractIoFilterChain.java:361)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1300(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(Ab stractIoFilterChain.java:659)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWrite(Abs tractIoFilterChain.java:587)

at org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(Ab stractIoFilterChain.java:361)

at org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(AbstractIo FilterChain.java:355)

at org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(SocketSessionImpl .java:166)

at org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:177)

at org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:168)

at org.jivesoftware.multiplexer.net.NIOConnection.deliver(NIOConnection.java:230)

at org.jivesoftware.multiplexer.ClientSession.deliver(ClientSession.java:254)

at org.jivesoftware.multiplexer.ServerPacketHandler.processRoute(ServerPacketHandl er.java:153)

at org.jivesoftware.multiplexer.ServerPacketHandler.handle(ServerPacketHandler.jav a:62)

at org.jivesoftware.multiplexer.ServerPacketReader$ProcessStanzaTask.run(ServerPac ketReader.java:128)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Please help me!!!

2

It seems that there is a memory leak somewhere. You should set "-XX:+HeapDumpOnOutOfMemoryError" You can specify also the directory with "-XX:HeapDumpPath=..." but the target directory must be writeable by the Openfire process.

The alternative is to use jmap to create heap dumps - jmap must be run as the same user Openfire is running. jmap and jhat are described for example here: http://petermodzelewski.blogspot.de/2013/06/short-jhat-tutorial-diagnosing.html

1 Like
3

Thanx LG for the reply…

Will enable the settings and collect memory dump and see if i can figure out where the leak is…

Meanwhile for 8000 ldap users with 2000-3000 online users, what is the best memory settings for openfire and connection manager ? … I dont mind keeping 20-30% extra. Can you please advise?

Thanks

4

Hi,

I had one more java out of memory today and I could generate the dump file. Below is the instance count generated through JHAT

Instance Counts for All Classes (including platform)

116835 instances of class java.util.HashMap$Entry

65899 instances of class [B

59965 instances of class java.util.TreeMap$Entry

47852 instances of class [I

33249 instances of class java.lang.Object

26156 instances of class java.math.BigInteger

25383 instances of class [C

19750 instances of class java.lang.String

14180 instances of class java.util.LinkedHashMap$Entry

13057 instances of class [Ljava.util.HashMap$Entry;

10982 instances of class [Ljava.lang.String;

10954 instances of class java.util.HashMap

10685 instances of class org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl

10685 instances of class org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1

10073 instances of class java.util.concurrent.ConcurrentLinkedQueue$Node

8596 instances of class java.lang.ref.ReferenceQueue$Lock

8596 instances of class java.util.LinkedHashMap

8594 instances of class java.lang.ref.ReferenceQueue

8562 instances of class sun.security.util.MemoryCache

8553 instances of class java.net.InetSocketAddress

8553 instances of class java.net.InetSocketAddress$InetSocketAddressHolder

8545 instances of class java.security.MessageDigest$Delegate

7369 instances of class java.util.concurrent.atomic.AtomicInteger

7032 instances of class [Ljava.lang.Object;

6600 instances of class java.util.ArrayList

6533 instances of class java.util.LinkedList

6423 instances of class java.util.concurrent.ConcurrentLinkedQueue

6418 instances of class [Ljava.security.cert.X509Certificate;

6408 instances of class sun.security.provider.MD5

4884 instances of class java.nio.HeapByteBuffer

4799 instances of class java.nio.DirectByteBuffer

4793 instances of class java.nio.DirectByteBuffer$Deallocator

4793 instances of class sun.misc.Cleaner

4584 instances of class sun.security.util.ObjectIdentifier

4401 instances of class sun.security.x509.AlgorithmId

4322 instances of class java.lang.ref.Finalizer

4282 instances of class java.net.InetAddress$InetAddressHolder

4280 instances of class sun.security.ssl.EphemeralKeyManager$EphemeralKeyPair

4280 instances of class sun.security.ssl.SSLSessionContextImpl

4280 instances of class sun.security.ssl.SunX509KeyManagerImpl$X509Credentials

4279 instances of class java.net.Inet4Address

4275 instances of class java.lang.StringBuilder

4273 instances of class sun.security.ssl.CipherBox

4273 instances of class sun.security.ssl.MAC

4272 instances of class com.sun.crypto.provider.ARCFOURCipher

4272 instances of class com.sun.crypto.provider.HmacCore

4272 instances of class com.sun.crypto.provider.HmacMD5

4272 instances of class javax.crypto.Cipher

4272 instances of class javax.crypto.Mac

3385 instances of class java.util.concurrent.ConcurrentHashMap$HashEntry

2676 instances of class [S

2539 instances of class sun.security.util.DerInputBuffer

Let me know if you can make out anything abnormal out of this? I would be very helpful if you can through some light on this

I can paste if you need any other out of JHAT

Many Thanks,

Shamjith

5

Unfortunately, there isn’t one actual Openfire class in your dump. Which seems very odd and will not help too much. Other than the mina classes, everything is from the JDK so that give little indiction of where your memory is being used.

Can you get a dump with memory usage instead of instance count?

6

new thread was started here: http://community.igniterealtime.org/thread/52007