java.lang.OutOfMemoryError (any idea how to fix?)

Spacegoose · November 5, 2011, 2:12pm

We are getting these every few days, have around 200 users. 3.7.1 Solaris 10.

Thanks,

Bill

here are the params:

INSTALL4J_ADD_VM_PARAMS="-Xms2048m -Xmx2048m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/apps/openfire/logs/gc.log -Dcom.sun.management.jmxremote.port=4848 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/apps/openfire/conf/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/apps/openfire/conf/jmxremote.access"

2011.11.02 20:49:48 org.jivesoftware.openfire.nio.ConnectionHandler -
java.lang.OutOfMemoryError
        at sun.security.pkcs11.wrapper.PKCS11.C_EncryptUpdate(Native Method)
        at sun.security.pkcs11.P11Cipher.implUpdate(P11Cipher.java:517)
        at sun.security.pkcs11.P11Cipher.engineUpdate(P11Cipher.java:464)
        at javax.crypto.Cipher.update(DashoA13*..)
        at com.sun.net.ssl.internal.ssl.CipherBox.encrypt(CipherBox.java:141)
        at com.sun.net.ssl.internal.ssl.OutputRecord.encrypt(OutputRecord.java:199)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:742)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:731)
        at com.sun.net.ssl.internal.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:98)
        at com.sun.net.ssl.internal.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:689)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:985)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:904)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:238)
        at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
        at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:893)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:632)
        at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:396)
        at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:334)
        at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:192)
        at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2694)
        at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:293)
        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:175)
        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:193)
        at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:136)
        at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:66)
        at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
        at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
        at javax.naming.InitialContext.init(InitialContext.java:223)
        at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:134)
        at org.jivesoftware.util.JiveInitialLdapContext.<init>(JiveInitialLdapContext.java:43)
        at org.jivesoftware.openfire.ldap.LdapManager.getContext(LdapManager.java:535)
        at org.jivesoftware.openfire.ldap.LdapUserProvider.loadUser(LdapUserProvider.java:112)
        at org.jivesoftware.openfire.user.UserManager.getUser(UserManager.java:234)
        at org.jivesoftware.openfire.user.UserNameManager.getUserName(UserNameManager.java:106)
        at org.jivesoftware.openfire.user.UserNameManager.getUserName(UserNameManager.java:87)
        at org.jivesoftware.openfire.roster.RosterItem.setSubStatus(RosterItem.java:252)
        at org.jivesoftware.openfire.roster.Roster.<init>(Roster.java:169)
        at org.jivesoftware.openfire.roster.RosterManager.getRoster(RosterManager.java:94)
        at org.jivesoftware.openfire.handler.PresenceUpdateHandler.broadcastUpdate(PresenceUpdateHandler.java:305)
        at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:147)
        at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:135)
        at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:199)
        at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:149)
        at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:85)
        at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84)
        at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java:355)
        at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaHandler.java:100)
        at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:272)
        at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:194)
        at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:169)
        at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:570)
        at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
        at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
        at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
        at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
        at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
        at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
        at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
        at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimpleProtocolDecoderOutput.java:58)
        at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:185)
        at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
        at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
        at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
        at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:239)
        at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:283)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
        at java.lang.Thread.run(Thread.java:662)
2011.11.02 21:16:51 org.jivesoftware.openfire.roster.Roster - Groups ([its-sts, its-fts]) include non-existent username (kma6)

Stefan_Reuter · November 5, 2011, 3:01pm

Set xmpp.pep.enabled to false in Openfire’s console to prevent the memory leak described in OF-439

Spacegoose · November 6, 2011, 12:26am

We already have this false.

Does the trace indicate PEP is the issue? I see SSL, LDAP, PKCS11, etc.

It happened again - as more users logged into the system this morning - is this related:

2011.11.07 09:36:31 org.jivesoftware.openfire.nio.ConnectionHandler -

what is nio?

Thanks,

Bill

Spacegoose · November 9, 2011, 1:37pm

I notice after a day the memory use becomes around half the heap and climbing - and doesn’t go down.

It usually goes up and down - but always settles again to around 3-400mb - in normal operations - or after a restsart (and stays that way until the following morning).

after 20 or so hours:

1309.69 MB of 2029.12 MB (64.5%) used

a lockup or crash usually ensues as the morning load (people logging in, etc.) starts …

the memory utilization climbs and does not come down as normal.

thinking of reverting to 3.6.4

thanks for any info,

Bill

Bill_MacAllister · November 10, 2011, 12:34am

Just thought I should note to say that I am seeing this problem as well.

Bill MacAllister

Bill_MacAllister · November 11, 2011, 5:43pm

Setting xmpp.pep.enabled to false resolved my issue.

Bill

Spacegoose · January 11, 2012, 8:07pm

We solved this issue a few weeks ago. The answer was simply to reduce the initial heap size.

We’d been using: -Xms2048m -Xmx2048m (which may have been default) -

changing -Xms2048 to -Xms256m appears to have solved the issue.

Thanks,

Bill

Kent1 · May 7, 2012, 11:59am

I’m having problem with changing those values…

If i change the value to e.g. -Xms1024m -Xmx2048m (in the openfireservice-vmoptions file), then i get the following error:

Robin_Collier · May 7, 2012, 4:51pm

There should be no space between the s and 1 in the first argument.

Kent1 · May 7, 2012, 7:50pm

True, and there isnt…

Robin_Collier · May 7, 2012, 9:38pm

Sorry

Kinda looked like there was on my monitor.

Mark_Dayanghirang · November 30, 2012, 12:19am

Try editing /etc/sysconfig/openfire

Set this to the path where openfire lives.

If this is not set the script will look for /usr/local/openfire, then

/opt/openfire.

#OPENFIRE_HOME=""

If there is a different user you would like to run openfire as,

change the following line.

#OPENFIRE_USER=“daemon”

If you wish to change the location of the openfire pid file,

change the following line.

#OPENFIRE_PIDFILE="/var/run/openfire.pid"

If you wish to explictly specific the location of the log directory,

you can set it here. Note that this applies to the logs generated outside

openfire itself. If you want to change the location of openfire’s own

logs, see the system property ‘log.directory’. If this is not set,

it will default to $OPENFIRE_HOME/logs.

#OPENFIRE_LOGDIR="/some/where/logs"

# If you wish to set any specific options to pass to the JVM, you can

# set them with the following variable.

OPENFIRE_OPTS="-Xms256m -Xmx512m"

If you wish to override the auto-detected JAVA_HOME variable, uncomment

and change the following line.

#JAVA_HOME=/usr/java/default

ps -ef | grep openfire

daemon 13763 1 13 19:07 ? 00:00:06 /opt/openfire/jre/bin/java **-server -Xms256m -Xmx512m **-DopenfireHome=/opt/openfire -Dopenfire.lib.dir=/opt/openfire/lib -classpath /opt/openfire/lib/startup.jar -jar /opt/openfire/lib/startup.jar

Hope this help.