Frequent disconnects - users being punted

I’m running into a problem with our openfire setup, where I’m noticing an excessive amount of RAM being used (on the order of 400mb+) and the following error messages which I’m a little confused about:

at org.jivesoftware.openfire.ldap.LdapVCardProvider.updateVCard(LdapVCardProvider. java:324)

at org.jivesoftware.openfire.vcard.VCardManager.setVCard(VCardManager.java:133)

at org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:8 3)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:49)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:349)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:101)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:311)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:276)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:175)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 885)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Thread.java:619)

2008.07.17 16:26:29 org.jivesoftware.openfire.ldap.LdapGroupProvider.populateGroups(LdapGroupProvide r.java:683)

java.lang.NullPointerException

2008.07.17 16:26:43 org.jivesoftware.openfire.ldap.LdapGroupProvider.populateGroups(LdapGroupProvide r.java:683)

java.lang.NullPointerException

2008.07.17 16:26:47 org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:92 )

java.lang.UnsupportedOperationException: LdapVCardProvider: Invalid vcard changes.

at org.jivesoftware.openfire.ldap.LdapVCardProvider.updateVCard(LdapVCardProvider. java:324)

at org.jivesoftware.openfire.vcard.VCardManager.setVCard(VCardManager.java:133)

at org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:8 3)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:49)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:349)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:101)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:311)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:276)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:175)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 885)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Thread.java:619)

2008.07.17 16:26:52 org.jivesoftware.openfire.ldap.LdapGroupProvider.populateGroups(LdapGroupProvide r.java:683)

java.lang.NullPointerException

2008.07.17 16:27:23 org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:92 )

java.lang.UnsupportedOperationException: LdapVCardProvider: Invalid vcard changes.

at org.jivesoftware.openfire.ldap.LdapVCardProvider.updateVCard(LdapVCardProvider. java:324)

at org.jivesoftware.openfire.vcard.VCardManager.setVCard(VCardManager.java:133)

at org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:8 3)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:49)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:349)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:101)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:311)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:276)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:175)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 885)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Thread.java:619)

2008.07.17 16:29:32 org.jivesoftware.openfire.ldap.LdapGroupProvider.populateGroups(LdapGroupProvide r.java:683)

java.lang.NullPointerException

2008.07.17 16:29:33 org.jivesoftware.openfire.ldap.LdapGroupProvider.populateGroups(LdapGroupProvide r.java:683)

java.lang.NullPointerException

2008.07.17 16:29:43 org.jivesoftware.openfire.ldap.LdapGroupProvider.populateGroups(LdapGroupProvide r.java:683)

java.lang.NullPointerException

2008.07.17 16:29:43 org.jivesoftware.openfire.ldap.LdapGroupProvider.populateGroups(LdapGroupProvide r.java:683)

java.lang.NullPointerException

2008.07.17 16:29:51 org.jivesoftware.openfire.ldap.LdapGroupProvider.populateGroups(LdapGroupProvide r.java:683)

java.lang.NullPointerException

2008.07.17 16:31:14 org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:92 )

java.lang.UnsupportedOperationException: LdapVCardProvider: Invalid vcard changes.

at org.jivesoftware.openfire.ldap.LdapVCardProvider.updateVCard(LdapVCardProvider. java:324)

at org.jivesoftware.openfire.vcard.VCardManager.setVCard(VCardManager.java:133)

at org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:8 3)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:49)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:349)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:101)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:311)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:276)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:175)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 885)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Thread.java:619)

2008.07.17 16:32:56 org.jivesoftware.openfire.ldap.LdapGroupProvider.populateGroups(LdapGroupProvide r.java:683)

java.lang.NullPointerException

2008.07.17 16:34:17 org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:92 )

java.lang.UnsupportedOperationException: LdapVCardProvider: Invalid vcard changes.

at org.jivesoftware.openfire.ldap.LdapVCardProvider.updateVCard(LdapVCardProvider. java:324)

at org.jivesoftware.openfire.vcard.VCardManager.setVCard(VCardManager.java:133)

at org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:8 3)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:49)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:349)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:101)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:311)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:276)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:175)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 885)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Thread.java:619)

— Marker inserted by bmenges at Jul 17, 2008 4:36:47 PM —

From what I can see in here, there are two messages; a failure on the vcard sync to ldap, and the ldap group population. I’m not sure what’s causing users to continously be kicked off the server, but these problems are also are a concern. Help and insight would be appreciated.

We run an LDAP authenticated against AD openfire 3.5.2 setup. Server is Debian 4.0r3. Stack/heap have 512/512mb ram, and the system is eating close to 90% of that now. Any ideas?

I’m having the same issue here (except memory usage is normal); using Openfire 3.50. I’ve tracked down the issue to a client intiated server error. I’m able to constantly reproduce it if I connect using Miranda IM with any avatar loaded. It will auto-boot anyone connected to the server using Pidgin (anything newer than 2.4.0), and Meebo clients. I have yet to have one single Spark client disconnect.

I want to keep vCards enabled, any suggestions???

Apparently judging from some other users on the forums this is a health check issue which is not yet supported by openfire (or so it seems). There’s apparently some kind of heartbeat check that bounces back and forth in newer clients (for example Pidgin 2.4.2 and above) that when used in conjunction with openfire 3.5.2 causes the connection to hard reset. The server then burns up a lot of memory re-establishing the connection.

Our solution was enabling parallel garbage collectors “-XX:+UseParallelGC” in /etc/defaults/openfire (though this is just a recommended update, I’m sure it will help with many problems beyond just this one) and xmpp.client.idle = -1

The xmpp.client.idle set to -1 basically ignores the heartbeats which would determine whether a client has lost connectivity. Instead normal messages will determine connectivity instead of the heartbeats. This fixed our problem.

mengesb wrote:

The xmpp.client.idle set to -1 basically ignores the heartbeats which would determine whether a client has lost connectivity. Instead normal messages will determine connectivity instead of the heartbeats. This fixed our problem.

Could you point out in the source code where your logic takes place? Just curious how you arrived at that conclusion.

thanks,

daryl

Don’t have time to go digging around in source for this problem being the above property that I’ve applied.

I’m more interested in developing a clustering solution for my openfire server first before exploring ping-of-death problems that I hope they fix in future JMs