My setup:
I just stood up 4 linux openfire servers within my environment with clustering enabled. Each machine has 1GB available to use at the moment and have a given range of 500-2000 sessions running at any given time. The plugins I have enabled are as followed.
Plugins:
Clustering Plugin
Monitoring Sevice (which does not seem to work for me as I am not getting the graphs, but that’s for later)
Presence Service
Registration
Search
User Service
User Status Plugin
The issue I am seeing is that the jvm memory is constantly ramping up and being purged by CG on a frequent basis (GC around every min or so). This to me does not look right? Also, there are random times where CPU spikes from 5% to 80%, sometimes for 1 min and others for 5 hours. This is very concerning to say the least as the only way I am able to alleviate the issue is to kick the openfire service and let is start fresh.
Has anyone ever experienced this type of issue before? If so, I’d love some insight on what could be going on. I’ve adjusted JVM setting to the best of my ability with no resolution.
Thanks
-Pat
Hello,
Are you able to get thread dumps (kill -3) from the java processes?
daryl
I’m also seeing this periodically on CentOS 5 x64. CPU usage goes to 100% and nobody can sign into chat, and I can’t get into the admin console because Openfire has used its allotted memory.
It has happened twice now since I upgraded to 3.7.0, and I’ve been running on 1 server successfully for a couple years now, 3.6.4 forever while there were no updates ran fine.
Daryl, I will try kill -3 next time it happens, thanks.
Hello,
I have given you bad guidance. You should probably get a heap dump. This can be accomplished with jmap, which is a part of the Java SDK. You’ll need to run jmap as the same user as the openfire process, so probably daemon. From root, you can run
su daemon -s /bin/sh -c 'jmap -dump:format=b,file=/tmp/heapdump <<pid>>'
where <> is the process ID of openfire.
daryl
Well my server is doing that thing right now! How convienient…
I don’t have the openjdk installed – will it mess with my openfire installation if I install it? I don’t really know jack about Java so I’m not sure if I install this if it will mess with the jre I have installed or what.
Thanks for your assistance, Daryl.
try installing java-1.6.0-sun-devel via yum install
daryl
That’s not available, but when I do yum provides */jmap, it says it’s in this package:
java-1.6.0-openjdk-devel-1.6.0.0-1.16.b17.el5.x86_64
Should I install that?
OK, Done!
Haha, now what do I do with this 250 MB file?
//edit1:
Ah, ok, found a gude:
http://blogs.sun.com/alanb/entry/heap_dumps_are_back_with
Thanks for getting me to the heapdump land, I’ll see if I can get his jhat thing working.
//edit 2:
Well, I jhat’d the file (wow, it used 2GB of ram…) and the web interface is up on port 7000.
I connect to it and I see: a buch of damn java jibberish!
Lol.
Any clue what I might be looking for specifically?
Thanks!
Message was edited by: George Alexander
Could you gzip the file and place on somewhere where I can download it?
daryl
you hit the PEP memory leak.
As a workaround, you can disable PEP, by setting the Openfire property xmpp.pep.enabled to false.
1 Like
Awesome, thanks!
Hopefully this solves my problem. Is this a documented/tracked bug? Can you link me so I can have a better understanding? How might I “hit” the bug?
Thanks for helping, Daryl!
This basically fixed itself once I upgraded to 3.7.0.