Help - Openfire Ramdonly disconnects everyone and needs a restart

Pedro_Reis · July 10, 2007, 9:22pm

Hi,

I’‘m evaluating openfire instead of going the communicator way, but I’‘m having a serious problem, I’'m running the latest version of Openfire and Spark on Unbuntu

Linux openfire 2.6.19-4-server #2 SMP Thu Apr 5 06:07:34 UTC 2007 i686 GNU/Linux

java version “1.6.0”

Java™ SE Runtime Environment (build 1.6.0-b105)

Java HotSpot™ Client VM (build 1.6.0-b105, mixed mode, sharing)

I started with the embebed database, but thinking it could be that i changed to mysql 5 but that didnt change anything.

My problem is that maybe 1 or 2 everyday all spark clients disconnect and then start on a loop of connecting and disconnecting, apart from this problem of everyone getting disconnected it’'s running fine.

Atm I’'m testing with 40 users, i have java set with 1GB of ram, this is to handle over 300 people and it doesnt seem to work well for 40 people

Here’‘s my server logs, I get this error about a strange character in a good few vcards, I dont know if it’‘s that that’'s crashing it but i dont see anything bad on the peopls vcards.

Without me being able to show this working well, there’‘s no chance of using openfire or even purchasing the enterprise edition, I would love to use openfire but I’'m at a loss.

error log, i have many like this one

2007.07.09 16:10:24 org.jivesoftware.openfire.vcard.DefaultVCardProvider.loadVCard(DefaultVCardProvi der.java:81) Error loadi

ng vCard of username: clara_trindade

org.dom4j.DocumentException: Error on line 1 of document : Character reference “&#0” is an invalid XML character. Nested exce

ption: Character reference “&#0” is an invalid XML character.

at org.dom4j.io.SAXReader.read(SAXReader.java:482)

at org.dom4j.io.SAXReader.read(SAXReader.java:365)

at org.jivesoftware.openfire.vcard.DefaultVCardProvider.loadVCard(DefaultVCardProv ider.java:76)

at org.jivesoftware.openfire.vcard.VCardManager.getOrLoadVCard(VCardManager.java:1 98)

at org.jivesoftware.openfire.vcard.VCardManager.getVCard(VCardManager.java:191)

at org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:1 07)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:48)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:300)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:104)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:67)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:289)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:153)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:132)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:200)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :266)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:326)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 885)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

at java.lang.Thread.run(Thread.java:619)

… etc

warn.log

2007.07.10 09:33:25 Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommend

ed.

2007.07.10 09:33:26 Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommend

ed.

I’'ll be gratefull for any help I could get.

Something I would love to see would be a Xen or Vmware image from Jive with a supported linux platform certified for Openfire.

Best Regards

Petr_Bouska · July 11, 2007, 3:22pm

I have the same problem with the same error. I’'am using Active Directory and ldapvcardavatar plugin. I tried uninstall this plugin and will see.

Bill_Asher · July 11, 2007, 6:07pm

I have the same problem. Running on Ubuntu 6.06 with about 400 people. Just crashes and have to restart, it seems like a memory leak issue, it has to be more stable than this, a Ubuntu issue?

LG1 · July 11, 2007, 7:24pm

Hi,

this is not a new issue, it was discussed in http://www.igniterealtime.org/forum/thread.jspa?messageID=146192 with little success. It may help to make sure that all vCards in the database are valid, especially the one of clara_trindade. Or do you get this error for all vCards?

LG

Cameron_Moore · July 11, 2007, 7:26pm

I take it your guys are running Openfire 3.3.2? Are you all using the ldapvcardavatar plugin? Where are you getting your vcard info – from LDAP/AD or direct from the Openfire DB?

Try turning on LDAP debugging (in the XML config, look for ldapDebugEnabled) and then turn on debugging in the admin interface. You may get some clues as to what part of the vcard XML structure is causing Openfire to blow up.

Pedro_Reis · July 11, 2007, 8:19pm

Hi,

thank you for your replies that was just a example, I have alot more for many different users, I have looked at their vcards and nothing special, name, department, telefone number, extention, email.

I’‘m using Mysql 5 (no ldap) everything on mysql, atm i’'m just using the “Search” plugin in effort to limit the amount of things to get better ods in finding what could be wrong.

I get many many erros like that one could that be the cause of the disconnects of all users? (the users arent disconnected when it gives those errors).

LG1 · July 11, 2007, 9:29pm

Hi,

the &#0 error sounds a lot like an invalid vCard entry. Is it possible for you to export a vCard which causes trouble and verify that there is no NULL character (0x00 or \u0000) in it?

LG

Pedro_Reis · July 11, 2007, 10:31pm

Hi,

I used the INSTR() of mysql to seach for it on the vcard table and it did not find any, I still dont know if this error would cause openfire to crash, I have copied the data from the vard table and emptied it, tomorrow I’'m going to get everyone to log into openfire 60+ people (the users created atm) and see what happens without vcards.

Oscar_Palomo · July 12, 2007, 6:03am

Hi.

I have the same disconnect problem.

I need to ask you to tell me if the vcard is the information entered by each user on the spark interface?

Regards,

Oscar

Pedro_Reis · July 12, 2007, 9:22am

Hi Oscar,

yes the VCard is the information entered by the user on spark on “Edit My Profile”.

LG1 · July 12, 2007, 1:52pm

Hi,

I and probably also Gato (the main developer) have a problem to reproduce this issue. Could someone post a vCard which fails? Hopefully a very short one without a photo so other users can import it in Openfire and check if it fails.

Does it help to clear the vCard cache on http://server:9090/system-cache.jsp ?

LG

Oscar_Palomo · July 12, 2007, 3:20pm

Pedro:

Thanks for the info. I am a bit confused where to identify the Vcards so I can send it to the forum.

I am having big problems with this issue. Yesterday I enabled the debug log and I am posting a part of it in here. I hope this helps.

2007.07.11 20:00:02

java.io.IOException: Connection reset by peer

at sun.nio.ch.FileDispatcher.read0(Native Method)

at sun.nio.ch.SocketDispatcher.read(Unknown Source)

at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)

at sun.nio.ch.IOUtil.read(Unknown Source)

at sun.nio.ch.SocketChannelImpl.read(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.j ava:232)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcesso r.java:206)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$500(SocketIoProce ssor.java:44)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:506)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:43)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

2007.07.11 20:06:04 Closing connection that has been idle: org.jivesoftware.openfire.nio.NIOConnection@1a119b3 MINA Session: (SOCKET, R: /168.234.238.212:1082, L: /168.234.238.110:5222, S: 0.0.0.0/0.0.0.0:5222)

2007.07.11 23:49:47 Ignoring extra content {}

2007.07.12 00:10:51 EOF

2007.07.12 00:16:48 Closing connection that has been idle: org.jivesoftware.openfire.nio.NIOConnection@12f15b0 MINA Session: (SOCKET, R: /190.56.108.119:22372, L: /168.234.238.110:5222, S: 0.0.0.0/0.0.0.0:5222)

2007.07.12 07:51:35

java.io.IOException: Broken pipe

at sun.nio.ch.FileDispatcher.write0(Native Method)

at sun.nio.ch.SocketDispatcher.write(Unknown Source)

at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)

at sun.nio.ch.IOUtil.write(Unknown Source)

at sun.nio.ch.SocketChannelImpl.write(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoProcesso r.java:428)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoProcesso r.java:366)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(SocketIoProce ssor.java:44)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:509)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:43)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

2007.07.12 08:07:14

java.io.IOException: Broken pipe

at sun.nio.ch.FileDispatcher.write0(Native Method)

at sun.nio.ch.SocketDispatcher.write(Unknown Source)

at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)

at sun.nio.ch.IOUtil.write(Unknown Source)

at sun.nio.ch.SocketChannelImpl.write(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoProcesso r.java:428)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoProcesso r.java:366)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(SocketIoProce ssor.java:44)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:509)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:43)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

2007.07.12 08:07:35

java.io.IOException: Broken pipe

at sun.nio.ch.FileDispatcher.write0(Native Method)

at sun.nio.ch.SocketDispatcher.write(Unknown Source)

at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)

at sun.nio.ch.IOUtil.write(Unknown Source)

at sun.nio.ch.SocketChannelImpl.write(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoProcesso r.java:428)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoProcesso r.java:366)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(SocketIoProce ssor.java:44)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:509)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:43)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

2007.07.12 08:08:30

java.io.IOException: Broken pipe

at sun.nio.ch.FileDispatcher.write0(Native Method)

at sun.nio.ch.SocketDispatcher.write(Unknown Source)

at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)

at sun.nio.ch.IOUtil.write(Unknown Source)

at sun.nio.ch.SocketChannelImpl.write(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoProcesso r.java:428)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.doFlush(SocketIoProcesso r.java:366)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(SocketIoProce ssor.java:44)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:509)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:43)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Please help I am in BIG trouble.

Regards.

Bill_Asher · July 13, 2007, 5:06pm

Mine crashes a few times a day and I’‘m not using vCard, just Ubuntu and MySQL. I can’'t figure out what the problem is… too many users ? About 400 or so…

Oscar_Palomo · July 13, 2007, 9:46pm

It200.

Hi. I know that you work a lot with Openfire. Do you have an advice on this problem? I am running just 60 users on this server.

Thanks in advance.

Oscar.

Petr_Bouska · July 16, 2007, 10:44am

HI,

I try put more description of my situation.

I’'am using OpenFire 3.3.2 on Windows Server 2003, embedded DB (now for testing) and users in AD.

I disabled ldapvcardavatar plugin and everything works fine for last few days. But I thing, that OpenFire isn`t looking in vCard now (so if there is the problem).

My situation before:

When one specific user tried connect some users was disconnected (and still, when they connected, they was disconnected). I had to clear caches and everything was fine. For this specific user is in log error (same like in first post).

I will try change DB a experiment with the record of user.

Oscar_Palomo · July 16, 2007, 3:04pm

Hi Pedro.

I am wondering if you got a solution for this problem. I made my users to erase their information from their profiles and also disabled the Private Data Storage feature.

I don’'t see any more replies so I am wondering if this is being taking care of by someone at ignite.

Thanks in advance for your help.

Oscar.

Oleksiy_Gayda · July 17, 2007, 5:20pm

I am having the same issue - everyone is getting disconnected anywhere from 3 to a dozen times a day. Solution anyone?

LG1 · July 17, 2007, 9:50pm

Hi oleksiyg, bash3r and Samuraj,

do you have the same vCard problems and are you using the same operating system, Java version, JVM parameters, Openfire version and database? If this is not the case you may want to create a new thread.

Please take a look at http://wiki.igniterealtime.org/display/WILDFIRE/JVMSettingsand+Debugging and enable the GC log, disable IPv6 if you don’'t use it and post if Openfire crashes (the JVM terminates in an unexpected way) or if it blocks - if it blocks you may want to get a Stacktrace.

It may be a serious problem as it seems that a lot of users have it but as long as the details are not clear (wrong JVM heap settings, JVM 1.6 or 1.5 crashes or blocking server, …) I wonder if you all have the exactly same issue.

LG