Openfire hang up

Openfire Openfire Support
1

Hello, about two weeks ago my openfire server 3.6.3 start to hang up few times a day, i didn’t understood in what problem was and did upgrade to 3.7.1. But server still continue to hang up. It uses all java memory and 33% of CPU.

I increase max java memory to 1024 mb, and maxconnection to 30. I use openfire server runing under windows 2003 sp2 with Microsoft SQL 2005 server runing on another computer. Average sever has 400-600 active sessions.

It hangs up always in same time from 1 p.m. to 3 p.m. every day (except weekends when only 20-30 active sessions on it).

Avereage java memory consumption is 100-200 Mb. And 1024 when it hangs up.

Can some one help me? I tried to disable PEP tried some other things from forum but nothing help me, have no idea what to do next.

Sorry for bad english.

Logs included.
logs.zip (7917 Bytes)

2

Every time same error and then server didn’t respond

2011.12.27 13:55:05 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.(Unknown Source)

at java.nio.ByteBuffer.allocate(Unknown Source)

at org.apache.mina.common.SimpleByteBufferAllocator.allocate(SimpleByteBufferAlloc ator.java:45)

at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:225)

at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:214)

at org.jivesoftware.openfire.nio.NIOConnection.deliver(NIOConnection.java:250)

at org.jivesoftware.openfire.session.LocalClientSession.deliver(LocalClientSession .java:843)

at org.jivesoftware.openfire.session.LocalSession.process(LocalSession.java:281)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToLocalDomain(RoutingTableI mpl.java:305)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:233)

at org.jivesoftware.openfire.roster.Roster.broadcastPresence(Roster.java:612)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:306)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:147)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:135)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:199)

at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:149)

at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:85)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84)

at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java: 355)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:100)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:272)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:194)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:169)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)

2011.12.27 13:55:52 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2011.12.27 13:56:32 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2011.12.27 13:56:37 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2011.12.27 13:56:12 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2011.12.27 13:55:55 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

Can someone help, please?

3

If you get the OutOfMemoryError always in “org.jivesoftware.openfire.session.LocalClientSession.deliver(LocalClientSessio n .java:843)” then you could try to catch this error and close the affected connection. This could work:

try {

conn.deliver(packet); // line 843

} catch (OutOfMemoryError e) {

conn.close()

}

4

Where should i insert this?

5

I too have a similar problem. I have open fire running on Windows XP. It used to hang up once a month or so and now it hangs up two or three times a day. Although I am not including any logs, the error states Java memory heap. Is there a way for me to increase the Java memory? Thank you all for your support.

6

Irving, please search the forum for openfire.vmoption or create a new thread.

Download the source and install the JDK which matches the JVM (1.6 or 1.5) of your Openfire server.

Open a “cmd” and replace “…path-to…” with real pathes which exist.

set CLASSPATH=..path-to..\openfire\lib\openfire.jar;..path-to..\openfire\lib\slf4j-log4j12.jar
cd ..path-to..\openfire\src\java\org\jivesoftware\openfire\session
javac LocalClientSession.java

This should create a new LocalClientSession.class file in the current directory.

Open “openfire.jar” and replace there the existing “org\jivesoftware\openfire\session\LocalClientSession.class” file with this one. Openfire should work as before as you did not really change anything. Take five minutes to test this, if you did use the wrong JDK you may get errors during startup.

Then edit this file and replace

@Override
    void deliver(Packet packet) throws UnauthorizedException {
        if (conn != null && !conn.isClosed()) {
                conn.deliver(packet);
      }
    }

with

@Override
    void deliver(Packet packet) throws UnauthorizedException {
        if (conn != null && !conn.isClosed()) {
            try {
                conn.deliver(packet); // line 843
            } catch  (OutOfMemoryError e) {
                int connID = conn.hashCode();
                try {
                    Log.error("1 - Closing connection " + connID + " from " + conn.getHostAddress());
                } catch (Exception e1) {
                }                                try {
                    conn.close();
                    Log.error("2 - Closing connection " + connID + " closed!");
                } catch (Exception e1) {                             }
                try {
                    Log.error("3 - Closing connection " + connID + " of  " + getUsername());
                } catch (Exception e1) {                             }
                try {
                    Log.error("4 - Closing connection " + connID + " packetlen=" + packet.toString().length());
                } catch  (OutOfMemoryError e2) {
                } catch (Exception e1) {                             }
                Log.error("5 - Closing connection " + connID + " because of OutOfMemoryError.");
            }                }
    }

Then compile and replace it again as described above. “javac” must not print errors and create a new “LocalClientSession.class” file.

If later an OOM error does occur the connection should be droppend and something should be logged. Of course that’s not what users really want, but maybe you have one user which is causing most of these problems.

7

Thanks for advice. I did as you say. Now i will wait till it hang up again. In which log file errors will appear?

8

Log.error() does usually log in the logs/error.log file as defined in the lib/log4j.xml file.

9

Server hangs at friday with this errors in log

2012.01.13 13:57:48 org.jivesoftware.openfire.IQRouter - Cannot reply an IQ error to another IQ error:

2012.01.13 14:00:50 org.jivesoftware.openfire.IQRouter - Cannot reply an IQ error to another IQ error:

2012.01.13 14:01:08 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 2818089 from 10.59.93.11

2012.01.13 14:01:08 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 18333261 from 10.59.61.21

2012.01.13 14:01:20 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 1742275 from 10.59.27.29

2012.01.13 14:01:23 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 16190388 from 10.59.42.23

2012.01.13 14:01:28 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 17005458 from 10.59.59.20

2012.01.13 14:01:37 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 1468386 from 10.59.6.34

2012.01.13 14:02:04 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 3501861 from 10.59.44.21

2012.01.13 14:02:09 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 15099932 from 10.59.61.23

2012.01.13 14:02:19 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 18166008 from 10.59.48.30

2012.01.13 14:02:30 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 5157692 from 10.59.93.111

2012.01.13 14:02:23 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 32556098 from 10.59.27.26

2012.01.13 14:02:48 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 3942310 from 10.59.112.28

2012.01.13 14:03:00 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 11693859 from 10.59.17.39

2012.01.13 14:03:14 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 18914277 from 10.59.113.85

2012.01.13 14:03:30 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 26422357 from 10.59.53.21

2012.01.13 14:03:56 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 19571855 from 10.59.26.21

2012.01.13 14:04:15 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 22755869 from 10.59.5.30

2012.01.13 14:05:29 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.13 14:05:56 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 20985815 from 10.59.8.41

And today with …

2012.01.16 13:53:53 org.jivesoftware.openfire.handler.PresenceSubscribeHandler - Internal server error

org.jivesoftware.openfire.user.UserNotFoundException: shishkova_aa@perm.frs.ru

at org.jivesoftware.openfire.roster.Roster.updateRosterItem(Roster.java:385)

at org.jivesoftware.openfire.handler.PresenceSubscribeHandler.manageSub(PresenceSu bscribeHandler.java:304)

at org.jivesoftware.openfire.handler.PresenceSubscribeHandler.process(PresenceSubs cribeHandler.java:158)

at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:189)

at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:85)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84)

at org.jivesoftware.openfire.roster.Roster.deleteRosterItem(Roster.java:459)

at org.jivesoftware.openfire.roster.Roster.deleteSharedUser(Roster.java:1062)

at org.jivesoftware.openfire.roster.RosterManager.groupUserDeleted(RosterManager.j ava:704)

at org.jivesoftware.openfire.roster.RosterManager.groupUserDeleted(RosterManager.j ava:662)

at org.jivesoftware.openfire.roster.RosterManager.memberRemoved(RosterManager.java :455)

at org.jivesoftware.openfire.event.GroupEventDispatcher.dispatchEvent(GroupEventDi spatcher.java:134)

at org.jivesoftware.openfire.group.Group$MemberCollection$1.remove(Group.java:439)

at java.util.AbstractCollection.remove(Unknown Source)

at org.jivesoftware.openfire.admin.group_002dedit_jsp._jspService(group_002dedit_j sp.java:293)

at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:530)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1216)

at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:11 8)

at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1187)

at org.jivesoftware.util.LocaleFilter.doFilter(LocaleFilter.java:74)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1187)

at org.jivesoftware.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingF ilter.java:50)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1187)

at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:78)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1187)

at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:164)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1187)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:425)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:494)

at org.eclipse.jetty.server.session.SessionHandler.handle(SessionHandler.java:182)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:93 3)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:362)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:867 )

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandler Collection.java:245)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.jav a:126)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)

at org.eclipse.jetty.server.Server.handle(Server.java:334)

at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:559)

at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.j ava:1007)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:747)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:209)

at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:406)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:4 62)

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)

at java.lang.Thread.run(Unknown Source)

2012.01.16 14:02:37 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 28871175 from 10.59.11.24

2012.01.16 14:03:07 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 11333348 from 10.59.112.60

2012.01.16 14:03:04 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 7686466 from 10.59.112.217

2012.01.16 14:02:45 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 1479409 from 10.59.112.128

2012.01.16 14:02:38 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 10153477 from 10.59.112.242

2012.01.16 14:04:48 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.16 14:04:02 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

2012.01.16 14:03:58 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 20083578 from 10.59.119.8

2012.01.16 14:03:46 org.jivesoftware.util.log.util.CommonsLogFactory - Housekeeping log.error( :

2012.01.16 14:03:22 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 2347943 from 10.59.9.20

2012.01.16 14:03:14 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 21995329 from 10.59.112.243

2012.01.16 14:05:35 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.16 14:05:40 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.16 14:05:57 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 13126744 from 10.59.2.25

2012.01.16 14:06:42 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.16 14:07:10 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

2012.01.16 14:08:04 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 16170362 from 10.59.17.36

2012.01.16 14:07:46 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 33247533 from 10.59.119.8

2012.01.16 14:09:23 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 10660835 from 10.59.55.29

2012.01.16 14:09:38 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

2012.01.16 14:10:19 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.16 14:10:18 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.16 14:11:15 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 21889555 from 10.59.113.31

2012.01.16 14:11:45 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.16 14:12:43 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

2012.01.16 14:13:04 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

2012.01.16 14:12:58 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

2012.01.16 14:12:51 org.jivesoftware.openfire.container.PluginManager - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.16 14:13:52 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

java.lang.OutOfMemoryError: Java heap space

2012.01.16 14:14:22 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 18690061 from 10.59.112.180

2012.01.16 14:14:54 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

2012.01.16 14:15:03 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

2012.01.16 14:14:54 org.jivesoftware.openfire.nio.ConnectionHandler - Java heap space

2012.01.16 14:15:10 org.jivesoftware.openfire.session.LocalClientSession - 1 - Closing connection 23177978 from 10.59.27.23

Every time server hangs - everyone is still online, but no one can’t recieve any message. Also processor time always used at 33% by openfire server0 (in windows task manager).

10

Looking at the output it seems that “conn.close()” does hang (probably somewhere in the NIOCOnnection class - a stack trace would help to see this detailed) and thus the connection is never closed.

Replacing “conn.close()” with “conn = null;” may help and cause other exceptions. As long as the server crashes it does not really matter.

11

You can also add these JVM parameters “-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=c:/tmp” and make sure that c: has enough free space (2 GB) - such a dump may get large.

If you have the JDK installed you may be able to use “jmap -histo OEPNFIRE-PID” and get some histograms of the heap … or it may fail as Openfire runs as a service and thus the normal user can not use it.

12

I will try to do histograms. Can you tell me where should i add JVM parameters? Should i add it to openfire-service.vmoptions ?

I also noticed that memory usage increasing during the time. It started from 300-400 mb and hangs when it reach 1024. But i disable PEP, and same problem was at 3/6/3 version, so i think there is one more memory leak. Is there any option which can temporaly disable any plugins on server without delete?

13

Put the JVM parameters in same file where you did put “-Xmx1G”, likely openfire-service.vmoptions.

On Windows “jmap” may fail as the service runs as different user, http://li-ma.blogspot.com/2010/08/how-to-monitor-java-app-started-as.html may help to still use jmap.