Wildfire crash!

Openfire Openfire Support
1

Hi guys.

I have installed Wildfire 3.2.4 on Windows 2000 Server.

I have about of 60 user, all the user aren’‘t concurrently, but the server en some times crash and I need to restart the server because it doesn’'t response.

I check the logs and see it:

[org.jivesoftware.wildfire.nio.ConnectionHandler.exceptionCaught(ConnectionHandl er.java:109)

]

java.lang.OutOfMemoryError: Java heap space

2007.04.13 18:31:33 [org.jivesoftware.wildfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:109)

]

java.lang.OutOfMemoryError: Java heap space

2007.04.13 18:31:55 [org.jivesoftware.wildfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:109)

]

java.lang.OutOfMemoryError: Java heap space

2007.04.13 18:54:15 [org.jivesoftware.wildfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:109)

]

java.lang.OutOfMemoryError: Java heap space

2007.04.13 18:57:46 [org.jivesoftware.wildfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:109)

]

java.lang.OutOfMemoryError: Java heap space

2007.04.13 21:37:04

I think that is a Java memory error, the server has 1GB of RAM, In the file vmoptions, I have:

-Xms256m

-Xmx512m

I have active the plug-ins:

  • Content Filter

  • Subsciption

In the Server Properties the memory always is between 80% and 90%.

Do you have any idea about this?

Regards.

2

I’'m having the same sort of issues, on Linux, with Wildfire having around 700 megs of memory.

Here’'s my file…

-Xms400m

-Xmx400m

-XX:+HeapDumpOnOutOfMemoryError

I recently reduced memory down to 400 and added the third line so that we can try to track down the memory leak. I’'m just waiting for it to crash at this point.

3

Hi,

http://www.igniterealtime.org/forum/thread.jspa?threadID=25528 describes a similar problem. Not sure which version fixes this problem and if this is really the same problem.

LG

4

The server crash again

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:121)

at org.jivesoftware.wildfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1493)

at org.jivesoftware.wildfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:161)

at org.jivesoftware.wildfire.nio.NIOConnection.close(NIOConnection.java:140)

at org.jivesoftware.wildfire.nio.NIOConnection.systemShutdown(NIOConnection.java:1 47)

at org.jivesoftware.wildfire.SessionManager.stop(SessionManager.java:1705)

at org.jivesoftware.wildfire.XMPPServer.shutdownServer(XMPPServer.java:863)

at org.jivesoftware.wildfire.XMPPServer.access$800(XMPPServer.java:90)

at org.jivesoftware.wildfire.XMPPServer$ShutdownHookThread.run(XMPPServer.java:815 )

2007.04.17 13:51:32 [org.jivesoftware.wildfire.container.PluginManager$PluginMonitor.run(PluginMana ger.java:909)

] Error unloading plugin gateway. Will attempt again momentarily.

2007.04.17 13:51:42 [org.jivesoftware.wildfire.container.PluginManager$PluginMonitor.run(PluginMana ger.java:909)

] Error unloading plugin gateway. Will attempt again momentarily.

2007.04.17 15:00:25 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:87)

] Line=19 The content of element type “dwr” must match “(init?,allow?,signatures?)”.

2007.04.17 16:26:28 [org.jivesoftware.wildfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:109)

]

java.lang.OutOfMemoryError: Java heap space

at java.util.Arrays.copyOfRange(Unknown Source)

at java.lang.String.(Unknown Source)

at org.dom4j.io.SAXContentHandler.characters(SAXContentHandler.java:299)

at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.characters(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocu ment(Unknown Source)

at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)

at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)

at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unkno wn Source)

at org.dom4j.io.SAXReader.read(SAXReader.java:465)

at org.dom4j.io.SAXReader.read(SAXReader.java:365)

at org.jivesoftware.wildfire.OfflineMessageStore.getMessages(OfflineMessageStore.j ava:179)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.initSession(PresenceUpd ateHandler.java:210)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:104)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:88)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:151)

at org.jivesoftware.wildfire.PresenceRouter.handle(PresenceRouter.java:123)

at org.jivesoftware.wildfire.PresenceRouter.route(PresenceRouter.java:69)

at org.jivesoftware.wildfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:75)

at org.jivesoftware.wildfire.net.StanzaHandler.processPresence(StanzaHandler.java: 306)

at org.jivesoftware.wildfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:231)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:153)

at org.jivesoftware.wildfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:131)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:200)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

5

Hi,

did you add -XX:+HeapDumpOnOutOfMemoryError to the JVM options? If you do you should take care that you have enough free space in your filesystem.

If you are using HTTP bind then this may be related to the problem mentioned in the other thread.

It would help a lot if one could offer a heapdump file of this crash for download, download-link via PM to Gato.

LG

6

http://prelive.zappos.com/java_pid29123.hprof.zip

That’'s my crash. Careful, it unzips to a 400 meg file…

In our case, I do not think that we are using HTTPbinding, only Fastpath on 3.2.4.

7

Hi,

I’‘ll take a look at it tomorrow. 400 MB is quite small but too big for my desktop so I’'ll use a server with enough free memory.

LG

8

Hi,

I see that the problem is caused by a lot of 4k objects which are allocated by nio/mina but I’'m not sure about the place where it starts happening. Not sure if SSLFilter if used for old-SSL and not for TLS - are you using port 5223 / old-SSL for pandion or other clients?

LG

Total of 728826 instances occupying 350669135 bytes.
162790 instances of class [B (339392535 bytes, a lot of byte[] 4069 objects)
155518 instances of class java.util.concurrent.atomic.AtomicInteger (622072 bytes)
155498 instances of class java.nio.HeapByteBuffer (5442430 bytes)
155476 instances of class org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer (2021188 bytes)
85963 instances of class java.util.concurrent.locks.ReentrantLock$NonfairSync (1375408 bytes)
81072 instances of class [Ljava.lang.Object; (150772 bytes) Object Tree:
org.apache.mina.transport.socket.nio.SocketSessionImpl
     java.util.concurrent.ConcurrentLinkedQueue           java.util.concurrent.ConcurrentLinkedQueue
               java.util.concurrent.ConcurrentLinkedQueue
                    ... (very often, object tree uses 300 MB)
               
Each java.util.concurrent.ConcurrentLinkedQueue object contains another java.util.concurrent.ConcurrentLinkedQueue object (except the last one) and:
     org.apache.mina.common.IoFilter$WriteRequest
          org.apache.mina.filter.SSLFilter$EncreyptedBuffer (containing one byte[] 4069 object)
9

All addresses 5222 Client to Server The standard port for clients to connect to the server. Connections may or may not be encrypted. You can update the security settings for this port.

All addresses 5223 Client to Server The port used for clients to connect to the server using the old SSL method. The old SSL method is not an XMPP standard method and will be deprecated in the future. You can update the security settings for this port.

All addresses 5269 Server to Server The port used for remote servers to connect to this server.

All addresses 9090 Admin Console The port used for unsecured Admin Console access.

All addresses 9091 Admin Console

I don’‘t use this server for anything other than Spark clients all using Fastpath (they connect via ssl). If there are better settings re: security for this, I’'m open to those suggestions!

10

Hi,

if you are using Spark then you are using TLS / port 5222.

See http://wiki.igniterealtime.org/display/NINJA/Monitorloadontheserver and install the stats plugin, it may be possible that a queue is filling up.

Add also “-XX:+PrintClassHistogram” to the JVM parameters, a kill -3 will then print out a class histogram with all java heap objects, their instance count and total space they occupy in the heap.

LG