Hey Everyone,
I know there are other posts around this issue and trust me I have read through them all. We have implemented a lot of the different solutions but just cannot seem to get rid of this issue. We are running the embedded database on a Windows 2003 server. It is a dedicated Openfire VM with 3GB of RAM. Our Java version is 1.6.0_18 Sun Microsystems Inc. – Java HotSpot™ Client VM. We have about 350 users using a combination of direct chat and MUC.
We have done the following to try and fix the problem:
-
Upgraded to 3.7.0
-
Set xmpp.pep.enabled = false
-
Adjusted the openfired.vmoptions high and low currently set at -Xms256m -Xmx768m
-
Removed all plugins
-
Adjusted Conversation settings, kicking idle users
-
For a previous issue we did change the embedded db from MEMORY to CACHED
I did add the monitoring plugin back to try to gather more specific information. The admin console shows 6-10% Java memory usage but the server is not stable and jams up after a day. We are getting this string of error messages followed by a bunch of the Java heap space errors over and over:
java.lang.NullPointerException
at org.hsqldb.persist.DataFileCache.get(Unknown Source)
at org.hsqldb.Table$RowStore.get(Unknown Source)
at org.hsqldb.Table.getRow(Unknown Source)
at org.hsqldb.DiskNode.findNode(Unknown Source)
at org.hsqldb.DiskNode.getRight(Unknown Source)
at org.hsqldb.Index.findFirstRow(Unknown Source)
at org.hsqldb.TableFilter.getFirstIterator(Unknown Source)
at org.hsqldb.TableFilter.findFirst(Unknown Source)
at org.hsqldb.Select.buildResult(Unknown Source)
at org.hsqldb.Select.getSingleResult(Unknown Source)
at org.hsqldb.Select.getResult(Unknown Source)
at org.hsqldb.CompiledStatementExecutor.executeSelectStatement(Unknown Source)
at org.hsqldb.CompiledStatementExecutor.executeImpl(Unknown Source)
at org.hsqldb.CompiledStatementExecutor.execute(Unknown Source)
at org.hsqldb.Session.sqlExecuteCompiledNoPreChecks(Unknown Source)
at org.hsqldb.Session.sqlExecute(Unknown Source)
at org.hsqldb.Session.sqlExecute(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
at org.hsqldb.jdbc.jdbcPreparedStatement.executeQuery(Unknown Source)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.logicalcobwebs.proxool.ProxyStatement.invoke(ProxyStatement.java:100)
at org.logicalcobwebs.proxool.ProxyStatement.intercept(ProxyStatement.java:57)
at $java.sql.Statement$$EnhancerByProxool$$db73d5bf.executeQuery()
at org.jivesoftware.openfire.vcard.DefaultVCardProvider.loadVCard(DefaultVCardProv ider.java:88)
at org.jivesoftware.openfire.vcard.VCardManager.getOrLoadVCard(VCardManager.java:2 39)
at org.jivesoftware.openfire.vcard.VCardManager.getVCard(VCardManager.java:232)
at org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:1 20)
at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:65)
at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:372)
at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:121)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:337)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:93)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:302)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:194)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:169)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.lang.OutOfMemoryError: Java heap space
Any help or direction on troubleshooting would be greatly appreciated!
Thanks!