Openfire 3.6.4 memory leak with Empathy

Openfire Openfire Support
81

Anyone?

82

Your patch did not patch cleanly against the 3.6.4 source. At least not for me.

83

jsu2,

Did you get the same error I mentioned in the Jira ticket? If so, you need to convert two of the files from CRLF to unix (dos2unix) and then try to apply the patch

daryl

84

I have committed my patch - all of the code is in SVN trunk right now. This should simplify getting things started considerably.

85

We are having serious memory leak issues that are NOT related to Empathy or pep. I started another thread for that incase anyone else is experiencing these issues: http://www.igniterealtime.org/community/thread/40791

Thanks

Daniel

86

Hi, Guus and All!

My configuration:

LDAP (AD) users and groups (around 120 users), rosters are distributed via shared groups, around six various plugins (Iā€™ve been trying disabling all of them while digging this problem, so the list is not important).

My environment:

Server (current): Openfire 3.7.0 release (started using with Openfire 3.6.4 release and further through some nightly builds and beta).

Clients (current): Miranda IM 0.9.19.0 (started using with Miranda IM 0.9.10.0 and further through stable releases).

Iā€™ve started suffering from daily memory runouts since putting 3.6.4 in production, Java memory configuration was default, min=max=64 MB. Then, Iā€™ve tried increasing memory to min=64 MB max=256 MB, but it extended total memory runout period only (to about 2 days). My temporary solution was scheduling a nightly server restart - and even if it was pretty acceptable in my (corporative) environment, it is not good for any server at all.

Iā€™ve read about Empathy/PEP/Java-memory some time ago, but: firstly, we donā€™t use Empathy at all; and secondly, it was not clear to me what actually PEP is. So, Iā€™ve been waiting for Openfire 3.7.0 release, hoping that it would solve my memory problem - and when 3.7.0 was finally released - and the problem was not solved - Iā€™ve started analyzing and digging actively. I must say, that Java Monitor is really a great tool for troubleshooting Java apps & servers (thank Guus for telling about it), it helped to see whatā€™s going on inside my server (and keeps telling me that).

When influence of all of my plugins has been excluded, Iā€™ve recalled this PEP thing, and have carefully read about it. Iā€™ve found out that it actually intercrosses with my other issue, but no one (not even guruā€™s ) have pointed that out to me. Now, I confirm that disabling PEP solved my long-suffered memory runouts (leak) problem, and here are a couple of related findings:

  1. I think that this problem is not related to Empathy only, I suppose it is related to any client, that supports PEP (i.e. sending/receiving Mood/Activity/Tunes information or other ā€œpersonal eventsā€) - and this aspect should be pointed out here or in some other place - so that people would be clearly aware of it. If ā€œpersonal eventsā€ are not actually being sent by clients (i.e. clients are not capable of doing so) - it doesnā€™t matter if PEP is enabled or not, the problem comes out only when PEP is actually used. And, until PEP realisation in Openfire is a known memory hog (until bugs are found and fixed) - wouldnā€™t making it disabled by default (out-of-box) be a good idea? I guess there are very few folks that use it consciously (and very few clients that support it).

  2. As far as I understand setting property xmpp.pep.enabled to false doesnā€™t actually disable PEP capability advertising by server, it simply disables processing of events of this given type - becase after setting this property and restarting server I still get (while connecting):

<iq type="result" id="****" from="****" to="****@****/****">
     <query xmlns="[http://jabber.org/protocol/disco#info](http://jabber.org/protocol/disco#info)">
          <identity category="server" name="Openfire Server" type="im" />
          <identity category="pubsub" type="pep" />
          <feature var="[http://jabber.org/protocol/pubsub#manage-subscriptions](http://jabber.org/protocol/pubsub#manage-subscriptions)" />

...

<feature var="[http://jabber.org/protocol/rsm](http://jabber.org/protocol/rsm)" />
     </query>
</iq>

So, is there a way to disable PEP-advertising (i.e. completely disable PEP)? If there is no such way - it would be desirable.

The point is that Miranda IM clients (I donā€™t know about other clients, including Empathy) show ā€œpersonal eventsā€ selecting/setting menus only when this capability is advertised by server, and if itā€™s not - there are simply no such menus (Iā€™ve found this out long ago, by trying to connecting to some public non-PEP non-Openfire servers). Thus, itā€™s not obvious to people why they have these menus and they canā€™t set ā€œpersonal eventsā€.

87

I figured out how to capture a memory leak report on the latest build (September 30, 2011). Memory usage was heavily concentrated in PEPService, TaskQueue, and the language modules.

http://community.igniterealtime.org/servlet/JiveServlet/previewBody/2207-102-1-2 490/leak_report_09302011.pdf

88

I can confirm that setting the property xmpp.pep.enabled set to false prevents a memory leak in openfire 3.7.

We had some folks using a few different clients (adium, pidgin, etc) and at least one of them was definitely causing a memory leak and weā€™d have to restart openfire at least once a week with a heap size of around 1 GB.

Now that weā€™ve put that property in place, itā€™s been smooth sailing ever since. BTW, we have 3 other servers where we tightly controlled which clients were allowed to connect and they did not suffer from the memory leak presumably because the client used didnā€™t use PEP features.

In my opinion, this is a MAJOR bug with openfire - a client program should not be able to take down your server by generating a memory leak. The server should correctly handle whatever the client does to prevent this.

Iā€™m rather surprised that this issue has persisted for so long and that it hasnā€™t been addressedā€¦

1 Like
89

It was addressed, but this project doesnā€™t how much manpower (developers) to do patches and this issue seems to be complicated one. Patches are welcome.

90

I believe that this is in fact fixed with the changes I made to pubsub about a month ago. The problem was in fact in pubsub, not PEP specifically.

I would encourage you to try the nightly build if at all possible as I would like have some help in verifying that it is indeed no longer an issue.

91

I want to report that the memory leak still exists on 3.7.1 and setting xmpp.pep.enabled=false seems to solve the problem.

92

Yes, that is already known. I was referring to the next version, the as of yet **unreleased **3.7.2, which is why I was mentioning that you would need to download the nightly build.

There are instructions related to that in the bottom of this thread. Unfortunately, the nightly build from the downloads page has not been updated since December, so that isnā€™t of any use.

93

Not sure if this will help anyone but I was able to fix memory issues I was having by pointing openfire to a different, 64 bit JRE. The OS is RHEL 6.1. Here is my config file from /etc/sysconfig/openfire.

JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre

OPENFIRE_OPTS="-Xms256m -Xmx1024m -Xss128k -Xoss128k -XX:ThreadStackSize=128 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:+PrintGCDetails -Xloggc:/opt/openfire/logs/gc.log"

I run openfire 3.7.0. Other than a few administrative changes yesterday, I havenā€™t had to restart Openfire in many many months after using that JRE. I keep the gc log enabled since it barely takes up any space (now that thereā€™s no memory errors ). That was how I was able to figure out where the problem was originally.