Java consumes too much memory

Myroslav · November 14, 2008, 2:36pm

Hello,

Could anyone hep me with following - my openfire server consumes too much memory.

Whyle now there is only:

Active Client Sessions: 21

and 3 chat rooms.

Java Version:
1.6.0_06 Sun Microsystems Inc. – Java HotSpot™ Server VM
Appserver:
jetty-6.1.x
Host Name:
vinci
OS / Hardware:
Linux / i386
Locale / Timezone:
en / Central European Time (1 GMT)
Java Memory

376.77 MB of 506.31 MB (74.4%) used

In coulple of daye it will consume 100% of memory so I need to restart it each several days.

Current Java VM parameters are:

-Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -XX:+PrintGCDetails -Xloggc:$/WF/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError

NB: Last 3 with -X I’ve added hoping to understand what is gooing on with server and they can’t be the cause of problem.

Martin_Weusten · November 14, 2008, 3:08pm

Probably there is no problem. You gave Java 512 MB, so it uses it. Java uses a garbage collector which only runs if memory is full.

When having so few users, you can smiply use the default settings, which is 64 MB.

Myroslav · November 14, 2008, 3:14pm

Then it’ll dye in several hours - there will be 100% filled red-color indicater and users will chaotically disconnect from server. Fact is - the only reason why I gave such large amount of memory to java was my wish to let server live a little bit longer.

andrew18 · November 14, 2008, 3:34pm

Hi,

Do you know how many messages are sent to the chat rooms? Do the chat rooms keep all history? Just asking because I had the same thing happening when doing a test with thousands of messages.

-t

Myroslav · November 14, 2008, 3:43pm

Not so many - there are 3 rooms, 2 of thea are empy and in one there is only 10-20 messages per hour and currently there are only 15 users in that room.

I keep history for last 100 messages - so as you can see server is slightly loaded.

guus · November 14, 2008, 4:35pm

Sounds like a nasty memory leak. I suspect that there are a couple of them in the Openfire core code, but nothing that should have this impact. What plugins are you using? Are you using custom plugins, or third party plugins? Do you have problems restarting one (or more) plugins properly?

Myroslav · November 14, 2008, 4:42pm

Here is a list of plugins we use: Broadcast (Jive Software), Monitoring Service (Jive Software), Registration (Ryan Graham), Search (Ryan Graham).

No custom plugins.

I’ve just tryed to resart all of them and didn’t notice any problems with it.

Philip1 · November 14, 2008, 5:16pm

We have exactly the same problem. We have about 100 User online and we use the Client Control, IM Gateway (with only MSN and ICQ), Monitoring Service and Search Plugins. I use a Mysql Database and Authenticate over our Active Directory. All users are shared to each other in seperate groups. If the memory usage reach 100% i get this error:

2008.10.22 09:41:02 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)]
java.lang.OutOfMemoryError: Java heap space

After i restart the service all Users reconnect and work like before. Openfire use at this time only 20MB but the usage increase permanently…

I try to disable all plugins but the memory usage grow up as before…

Everytime I get some error messages like this:

2008.10.22 09:41:49 [org.jivesoftware.openfire.roster.Roster.(Roster.java:177)] Groups ([IT Support]) include non-existent username (info)

And:

2008.10.22 09:32:07 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)]
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at org.apache.mina.filter.support.SSLHandler.messageReceived(SSLHandler.java:313)
at org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java:392)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.support.AbstractIoFilterChain$HeadFilter.messageReceived (AbstractIoFilterChain.java:499)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.fireMessageReceived(Abstra ctIoFilterChain.java:293)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.j ava:228)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcesso r.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProce ssor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

One crash was early in the morning where definitely nobody was online. So its very strange that the memory increase but nobody doing something. We use the server only in our internal network.

Myroslav · November 14, 2008, 5:23pm

It seems like our error, but we’ve got other error messages:

2008.11.14 17:50:09 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)]
java.lang.ArrayIndexOutOfBoundsException

But we use embedded db.

So for the moment the only one solution I’ve got is - to use more stress-stable client software, and restart opernfire every 2-3 days.

guus · November 14, 2008, 5:26pm

Sounds like a pretty vanilla setup.

It would be interesting to generate a heapdump of a Openfire suffering from the problem. That way, you’d be able to analyze it in a tool like JProfiler or JHat. I’m not sure if you’re in a position to create one?

A quick google showed at least two ways of obtaining a heapdump:

Start the application with HPROF enabled: java -agentlib:hprof=file=file,format=b application; on Control-Break, VM generates a heap dump.
Start VM with -XX:+HeapDumpOnCtrlBreak; on Ctrl-Break, VM generates a heap dump. (Introduced in Java SE 5.0 update 14.)

Gaston_Dombiak · November 14, 2008, 5:34pm

The way I like to get heapdumps when running out of memory is by adding -XX:+HeapDumpOnOutOfMemoryError. That setting will automatically create a heap dump when Java runs out of memory. I would be very interested in seeing your heap dump. We have been using Openfire on jivesoftware.com for more than 3 months without a single restart. IOW, I’m curious.

– Gato

guus · November 14, 2008, 5:37pm

Please reduce the total size of the heap if you do use ‘on OOM’. Trying to work with a heapdump that’s 128MB is easier than working with one that’s 1024MB

Myroslav · November 14, 2008, 5:38pm

great! I’m going to send it to you after I’ll get it.

LG1 · November 16, 2008, 3:19pm

Hi,

did you create some thread dumps? To create a stack trace use ´jstack openfire-pid >/tmp/javacore.txt´ and make sure to use different names if you create more than one stack trace. A quick way to count the used threads is ´ps -T -p $pid_of_openfire | wc -l´

http://www.igniterealtime.org/community/thread/35959 could be related to your problem. In any case it would be nice to know that you do not have a similar problem.

You should use a valid path within “-Xloggc:$/WF/logs/gc.log” like “-Xloggc:/tmp/gc.log” otherwise the info will go to STDERR (likely nohup.out).

LG

Philip1 · December 12, 2008, 12:43pm

whats going on? Have you find a solution for your problem? I still have to restart openfire every weekend to prevent a crash. I updated allready to 3.6.2 with no problems but the problem is still there. Currently we’re planning to provide it to all of our employees so it grow up from 100 users up to 250 over the whole world. So i have no free time to restart the service because then every time users have to use it. It would be very nice if we can solve the problem. I try to start some extra debug info and so on.

regards

Lunks

Nicolyn_Parks · December 12, 2008, 5:18pm

Hi,

I am having the exact problem. My memory usage is 99.7%!!! I’m restarting Openfire every 2 hours as users cannot login. Could someone please help me>!?!?!? I’m not a programmer, so are there any other solutions to this problem? I can’t even log inot the Admin Console. Thanks.

Gaston_Dombiak · December 12, 2008, 5:45pm

First off, try running the server with no plugins installed. If the server works fine then add the plugins back one at a time and make sure that the server keeps running fine. Eventually you should identify which plugin was causing this problem.

If you are running with no plugins then most likely (if you are running 3.6.2) is that some client is flooding the server with messages? In this case, I would recommend using the load stats plugin to collect information about the MINA queues/buffers. Finally, you should add -XX:+HeapDumpOnOutOfMemoryError to the list of command line parameters when starting Openfire. That new parameter will instruct Java to generate a dump of the memory and store it in a file. This file can then be analyzed to identify what was consuming so much memory.

– Gato

Nicolyn_Parks · December 12, 2008, 7:46pm

HI!!!

Thank you so much for responding to my cry for help!!! I’ve deleted all the

plugins and I’m still running out of memory. There are 2 processes running:

openfire.exe that’s avg. about 6,908K worth of CPU Memory and openfired.exe

that’s averaging a whopping 112,816K worth of memory. I believe the second

one is the problem. Since we’ve upgraded to 3.6.2, we have been having a

lot of problems with memory. I do believe that I need to run the

HeapDumpMemoty command you sent me, but could you tell me EXACTLY where to

go to add this to the command line and do I need to add the “**”. And where

do I go to get the load stats plugin?? Is this the same as the monitoring

plugin? Thanks so much again!!

Regards,

Nicolyn C. Parks

Business Systems Support

IT Department

Belize City, Belize

Phone: 011-501-222-4757 ext. 274

Fax: 011-501-222-4759

CONFIDENTIALITY NOTICE

The information contained in this message and any attachments are intended

solely for the use of the individual or entity to whom they are addressed.

It may contain privileged or confidential information. If you are not the

intended recipient you must not copy, distribute or take any action in

reliance on it. If you have received this e-mail in error please notify the

sender and delete the email. Thank You.

LG1 · December 12, 2008, 8:20pm

Hi Nicoly,

http://www.igniterealtime.org/builds/openfire/docs/latest/documentation/install- guide.html reads

Custom Parameters

Advanced users may wish to pass in parameters to the Java virtual machine (VM) to customize the runtime environment of Openfire. You can do this by creating vmoptions files in the bin/ directory of your Openfire installation. For the Windows service, you’d create a new text file called openfire-service.vmoptions. Each parameter to the VM should be on a new line of the file. For example, to set the minimum heap size to 512 MB and max VM heap size to 1024 MB, you’d use:

-Xms512m
-Xmx1024m

To create parameters for the normal launcher, create a file called openfired.vmoptions (since the openfire.exe launcher invokes the openfired.exe executable to actually start the server).

So you may want to increase the Java memory to 256, 512 or 1024 MB by adding a line like

-XmX512m
to openfired.vmoptions.

LG

Nicolyn_Parks · December 12, 2008, 10:22pm

Hi LG,

That seemed to have helped. Can I just confirm the name of the file??

Should it be openfired.vmoptions or openfired.vmoptions.txt?? Also, the

little meter that shows the Java Memory is still reading 99.3%. How do I

decrease that? Also, how do I decrease the size of the Current Search

Index, the Archived Message Count and the Archived Conversation Count??

Thanks.