Out of Memory : Help needed

Dear Folks,

We are facing regular ‘Out Of Memory’ exceptions in our openfire set up from the last one month (may be due to increase of load). The details are as given below.

Please let us if you have any inputs regarding this. Thanks

Details

=======

  1. Various types of OutOfMemory exceptions faced :

Type 1 >>

java.lang.OutOfMemoryError

at sun.misc.Unsafe.allocateMemory(Native Method)

at java.nio.DirectByteBuffer.(DirectByteBuffer.java:101)

at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)

at org.mortbay.io.nio.NIOBuffer.(NIOBuffer.java:51)

at org.mortbay.jetty.nio.AbstractNIOConnector.newBuffer(AbstractNIOConnector.java: 55)

at org.mortbay.jetty.AbstractBuffers.getBuffer(AbstractBuffers.java:117)

at org.mortbay.jetty.security.SslSelectChannelConnector.getBuffer(SslSelectChannel Connector.java:124)

at org.mortbay.jetty.security.SslHttpChannelEndPoint.(SslHttpChannelEndPoint .java:60)

at org.mortbay.jetty.security.SslSelectChannelConnector.newEndPoint(SslSelectChann elConnector.java:502)

at org.mortbay.jetty.nio.SelectChannelConnector$1.newEndPoint(SelectChannelConnect or.java:108)

at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:533)

at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:166)

at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java :124)

at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)

Type 2 >>

Exception in thread “timer-fastpath” java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method)

at java.lang.Thread.start(Thread.java:640)

at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExe cutor.java:727)

at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)

at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.jav a:78)

at org.jivesoftware.openfire.fastpath.util.TaskEngine$TimerTaskWrapper.run(TaskEng ine.java:312)

at java.util.TimerThread.mainLoop(Timer.java:512)

at java.util.TimerThread.run(Timer.java:462)

Type 3 >>

A fatal error has been detected by the Java Runtime Environment:SIGSEGV (0xb) at …

Exceptions::throw_stack_overflow_exception(Thread*, char const*, int)+0xf2

  1. Configuration details are as follows

Openfire version used : 3.6.4 with Fastpath version 4.1.0

OS : Linux 2.6.18-348.1.1.el5 x86_64

JVM : 32 bit 1.6.0_22 version

  1. The load on system is as follows :

45000 chats per day with around 1500 concurrent chats at any given point of time

We resolve this issue with a file named: openfired.vmoptions.

Place this archive on the folder bin.

The content of file is:

-Xms512m

-Xmx1024m

First line: initial memory of openfire

Second Line: max memory of openfire

PS.: Sorry for my english…

We use Windows Server 2008 R2…

Message was edited by: Wellington Wilczak Pereira

Thanks for your suggestion.

We have actually set these parameters as follows -Xms2048m -Xmx2048m -XX:PermSize=256m -XX:MaxPermSize=256m -Xss256k.

And thats where we are stuck because decreasing Xms is resulting in frequent Full GC sweeps and a potential OutOfMemory due to Heap size and setting it at 2G is resulting in low process memory (C-space) resulting in OutOfMemory - unable to create new native thread.

Appreciate if you have any findings on how to tune Openfire and minimise OutOfMemory exceptions.

Thanks,

Sudhir

How much memory is on the physical server? You cannot assign more memory to Java than your system actually has… (of course!). How many users? Does your server has swap space turned on (if linux/unix) or pagefile (if windows)?

Actually I see you already provided those details! Whoops!

I still ask though, how much physical memory does teh server have on it?


The exceptions being thrown are coming from java.nio.* classes, which typically work with system memory/buffers instead of java ones (ByteBuffer.allocateDirect() for example makes a buffer in system memory to try to avoid having to bring that data into the JVM heap, usually for performance reasons). THis may indicate your system is running out of memory totally, including the OS and everything else.

When you start to have these problems, on the server, see if your can do a:

~]# free -m

and paste the output here. I’m betting your system has started to swap, or if swap is off, it is then hard-crashing and the kernel is trying to kill off processes so that it doesn’t crash (the system). Just a guess though…

Also, try leaving the XX:PermSize and the other optiosn out of the openfire.vmoptions file. Just try setting the Xms and Xmx settings and leave the JVM to determine the rest. See if that may help a little.

Setting Xss to 256k may be a bad idea. The threads need more memory and OOMs are more likely.

Setting “-Xss128k -Xoss128k -XX:ThreadStackSize=128” may work, so the threads will use less memory.

Running a 32-bit JVM with “-Xms2048m” limits the available native memory for the java process. You need to decrease the Xmx value if you get more native OOMs (“unable to create new native thread”).

Do you use the embedded or an external database?

Thank you folks for your replies.

Few clarification :

  1. We set Xmx and Xms after a long gc pattern monitor and as a fix for OutofMemory due to Heapsize limitations

  2. It was fine for 3 months after we did point 1 but off late we started getting Stack size issues (Native memory OOM). We understand that this is because of low C space (as C space = 4G - [Perm Gen + Young space + Tenured]).

So this is the chicken and egg problem we got involved in. Low Heap -> OOM due to Heap size; Low Stack --> OOM due to native thread…:frowning:

For now we have resorted to daily restart of openfire which has prevented OOMs

Btw with respect to Swap space, currently the swap free is 16386 and swap utilized is 0. However this is only because we are doing daily restarts of openfire. Before that the swap used to be completely full. We were not able to identify the culprit objects which were resulting in memory leaks or were holding up memory and not getting garbage collected.

Two questions :

  1. Is there any known issues about particular classes/ objects in openfire which holds up memory and doesnt get garbage collected eventually moving to tenured space ? [This may explain why daily restarts is solving OOMs).

  2. Most of the OOMs related to Native thread were in timer-fastpath. What exactly is happening ? Is there any known issues or have we done any wrong configuration for this ? Kindly advise

Thanks,

Sudhir

I’ve suspected for awhile there is something wrong with FastPath but I have never been able to prove it. We had 500-800 fathpath chats a day going on and when we had this Openfire would die 3-4 days the web interface would stop working. We do maybe 100 a day now or less and we last 12-14 days now.

2. Most of the OOMs related to Native thread were in timer-fastpath. What exactly is happening ? Is there any known issues or have we done any wrong configuration for this ?

I assume that fastpath creates often new threads instead of using a thread pool. This would explain why the OOMS do occur here. One could look at the source code or use a JMX client and monitor ThreadMXBean.getTotalStartedThreadCount().

Yes, I have also observed that fastpath creates new threads instead of using a pool. Currently we are doing profiling to get details. Will keep you all posted. Meanwhile if anyone has experienced these issues and have a fix, please let us know. Thanks !!

My company is testing out Fastpath at the moment, and may scale it out if it’s a successful run. It’d be nice to pick this project back up again (and fix some of these things like thread pooling)… although I don’t have a ton of spare time anymore…

Just moments ago my web interface stop responding, I went to Fastpath and Tools to bring up the queue chats and I got this.

Once I restarted the server it all went back to normal.

HTTP ERROR 500

Problem accessing /webchat/. Reason:

No Java compiler available

Caused by:

java.lang.IllegalStateException: No Java compiler available

at org.apache.jasper.JspCompilationContext.createCompiler(JspCompilationContext.ja va:224)

at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:559)

at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:303)

at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)

at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:547)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:480)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:520)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:22 7)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:94 1)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:409)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186 )

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:875 )

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:288)

at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:115)

at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:556)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:547)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1359)

at org.jivesoftware.webchat.SetCharacterEncodingFilter.doFilter(SetCharacterEncodi ngFilter.java:45)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.jivesoftware.webchat.SetupFilter.doFilter(SetupFilter.java:92)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1330)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:478)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:520)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:22 7)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:94 1)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:409)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186 )

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:875 )

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandler Collection.java:250)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.jav a:149)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)

at org.eclipse.jetty.server.Server.handle(Server.java:349)

at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:441)

at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConne ction.java:919)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:582)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:51 )

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.jav a:586)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java :44)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598 )

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)

at java.lang.Thread.run(Unknown Source)

Powered by Jetty://