powered by Jive Software

Openfire 3.6.4 memory leak with Empathy

#74

I’ve been experiencing memory utilization problems as well: http://www.igniterealtime.org/community/message/197613#197613

I’ve seen usage of Empathy go up and instead of running out of memory once a month, it’s almost daily now. I can make my heap dumps available to the developers if they are interested in profiling it.

#75

The fix for http://bugs.freedesktop.org/show_bug.cgi?id=21151 (Should only query SOCK5 proxies when needed) has made it into Ubuntu Jaunty-Proposed repositories now.

It’ll stop the random servers being called from Empathy, although doesn’t seem related to Guus’s PEP theory.

To enable Proposed updates in Ubuntu, goto “Software Sources” in Administration, goto the updates tab, and selected “Proposed”. Then reload your sources. telepathy-gabble should then appear in update-manager.

#76

Couple Empathy clients connected, no problem so far, the memory use doesn’t show any suddent increase like before. I guess u did it Guus, great job! I will try to add some more Empathy clients to see if it’s stable.

#77

Hi Guus,

Are there plans for this to be fixed in the next Openfire release (and if so is there a timescale on this)? Or will we have to stick with disabling the PEP?

Thanks for your help.

#78

Hi Dave,

Sorry that it has taken so long to get back to you.

I’ve been trying to identify the cause of the problem, but so far, have been unable to do so. I’ve asked other developers to have a look too, but none of them have been successful either. Sadly, my attempts are severely hindered by lack of time - I’m doing this in my spare time, which is limited.

I’m not comfortable at all releasing the next version of Openfire with this bug in it, but there’s going to be a point in the near future where I feel we should be pragmatic, and move forward. The release has been postponed for to long now.

In the meantime, the lead developer of the Empathy client has confirmed that updates have been released that should dramatically reduce the impact of the bug. Thanks, Sjoerd! I haven’t tested the new client yet though.

Although I’m having trouble identifying the exact cause, I did manage to make some progress. While investigating, I’ve discovered a number of smaller and bigger issues in the PEP / pubsub routines of Openfire. There’s no obvious direct link between these issues and the memory leak that we’ve been discussing here, but the issues that I’m addressing now are likely candidates, in the sense that these kind of (concurrency-related) bugs are known to cause these kind of problems. I’m currently busy rewriting parts of the PEP routines. I’m hoping that my general improvements make this problem go away (or at least help me to identify the cause). This borders on the edges of educated guessing and wishful thinking, but hey, it’s the holiday season.

Regards,

Guus

#79

Thanks for the detailed update Guus. Much appreciated. If I could buy you a beer I would

Have a great holiday

#80

I have attached a patch that should fix this (and related) problems in the JIRA issue (OF-82). I’d be greatful if someone would give it a test.

#81

Anyone?

#82

Your patch did not patch cleanly against the 3.6.4 source. At least not for me.

#83

jsu2,

Did you get the same error I mentioned in the Jira ticket? If so, you need to convert two of the files from CRLF to unix (dos2unix) and then try to apply the patch

daryl

#84

I have committed my patch - all of the code is in SVN trunk right now. This should simplify getting things started considerably.

#85

We are having serious memory leak issues that are NOT related to Empathy or pep. I started another thread for that incase anyone else is experiencing these issues: http://www.igniterealtime.org/community/thread/40791

Thanks

Daniel

#86

Hi, Guus and All!

My configuration:

LDAP (AD) users and groups (around 120 users), rosters are distributed via shared groups, around six various plugins (I’ve been trying disabling all of them while digging this problem, so the list is not important).

My environment:

Server (current): Openfire 3.7.0 release (started using with Openfire 3.6.4 release and further through some nightly builds and beta).

Clients (current): Miranda IM 0.9.19.0 (started using with Miranda IM 0.9.10.0 and further through stable releases).

I’ve started suffering from daily memory runouts since putting 3.6.4 in production, Java memory configuration was default, min=max=64 MB. Then, I’ve tried increasing memory to min=64 MB max=256 MB, but it extended total memory runout period only (to about 2 days). My temporary solution was scheduling a nightly server restart - and even if it was pretty acceptable in my (corporative) environment, it is not good for any server at all.

I’ve read about Empathy/PEP/Java-memory some time ago, but: firstly, we don’t use Empathy at all; and secondly, it was not clear to me what actually PEP is. So, I’ve been waiting for Openfire 3.7.0 release, hoping that it would solve my memory problem - and when 3.7.0 was finally released - and the problem was not solved - I’ve started analyzing and digging actively. I must say, that Java Monitor is really a great tool for troubleshooting Java apps & servers (thank Guus for telling about it), it helped to see what’s going on inside my server (and keeps telling me that).

When influence of all of my plugins has been excluded, I’ve recalled this PEP thing, and have carefully read about it. I’ve found out that it actually intercrosses with my other issue, but no one (not even guru’s ) have pointed that out to me. Now, I confirm that disabling PEP solved my long-suffered memory runouts (leak) problem, and here are a couple of related findings:

  1. I think that this problem is not related to Empathy only, I suppose it is related to any client, that supports PEP (i.e. sending/receiving Mood/Activity/Tunes information or other “personal events”) - and this aspect should be pointed out here or in some other place - so that people would be clearly aware of it. If “personal events” are not actually being sent by clients (i.e. clients are not capable of doing so) - it doesn’t matter if PEP is enabled or not, the problem comes out only when PEP is actually used. And, until PEP realisation in Openfire is a known memory hog (until bugs are found and fixed) - wouldn’t making it disabled by default (out-of-box) be a good idea? I guess there are very few folks that use it consciously (and very few clients that support it).

  2. As far as I understand setting property xmpp.pep.enabled to false doesn’t actually disable PEP capability advertising by server, it simply disables processing of events of this given type - becase after setting this property and restarting server I still get (while connecting):

<iq type="result" id="****" from="****" to="****@****/****">
     <query xmlns="[http://jabber.org/protocol/disco#info](http://jabber.org/protocol/disco#info)">
          <identity category="server" name="Openfire Server" type="im" />
          <identity category="pubsub" type="pep" />
          <feature var="[http://jabber.org/protocol/pubsub#manage-subscriptions](http://jabber.org/protocol/pubsub#manage-subscriptions)" />

...

<feature var="[http://jabber.org/protocol/rsm](http://jabber.org/protocol/rsm)" />
     </query>
</iq>

So, is there a way to disable PEP-advertising (i.e. completely disable PEP)? If there is no such way - it would be desirable.

The point is that Miranda IM clients (I don’t know about other clients, including Empathy) show “personal events” selecting/setting menus only when this capability is advertised by server, and if it’s not - there are simply no such menus (I’ve found this out long ago, by trying to connecting to some public non-PEP non-Openfire servers). Thus, it’s not obvious to people why they have these menus and they can’t set “personal events”.

#87

I figured out how to capture a memory leak report on the latest build (September 30, 2011). Memory usage was heavily concentrated in PEPService, TaskQueue, and the language modules.

http://community.igniterealtime.org/servlet/JiveServlet/previewBody/2207-102-1-2 490/leak_report_09302011.pdf

#88

I can confirm that setting the property xmpp.pep.enabled set to false prevents a memory leak in openfire 3.7.

We had some folks using a few different clients (adium, pidgin, etc) and at least one of them was definitely causing a memory leak and we’d have to restart openfire at least once a week with a heap size of around 1 GB.

Now that we’ve put that property in place, it’s been smooth sailing ever since. BTW, we have 3 other servers where we tightly controlled which clients were allowed to connect and they did not suffer from the memory leak presumably because the client used didn’t use PEP features.

In my opinion, this is a MAJOR bug with openfire - a client program should not be able to take down your server by generating a memory leak. The server should correctly handle whatever the client does to prevent this.

I’m rather surprised that this issue has persisted for so long and that it hasn’t been addressed…

1 Like
#89

It was addressed, but this project doesn’t how much manpower (developers) to do patches and this issue seems to be complicated one. Patches are welcome.

#90

I believe that this is in fact fixed with the changes I made to pubsub about a month ago. The problem was in fact in pubsub, not PEP specifically.

I would encourage you to try the nightly build if at all possible as I would like have some help in verifying that it is indeed no longer an issue.

#91

I want to report that the memory leak still exists on 3.7.1 and setting xmpp.pep.enabled=false seems to solve the problem.

#92

Yes, that is already known. I was referring to the next version, the as of yet **unreleased **3.7.2, which is why I was mentioning that you would need to download the nightly build.

There are instructions related to that in the bottom of this thread. Unfortunately, the nightly build from the downloads page has not been updated since December, so that isn’t of any use.

#93

Not sure if this will help anyone but I was able to fix memory issues I was having by pointing openfire to a different, 64 bit JRE. The OS is RHEL 6.1. Here is my config file from /etc/sysconfig/openfire.

JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre

OPENFIRE_OPTS="-Xms256m -Xmx1024m -Xss128k -Xoss128k -XX:ThreadStackSize=128 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:+PrintGCDetails -Xloggc:/opt/openfire/logs/gc.log"

I run openfire 3.7.0. Other than a few administrative changes yesterday, I haven’t had to restart Openfire in many many months after using that JRE. I keep the gc log enabled since it barely takes up any space (now that there’s no memory errors ). That was how I was able to figure out where the problem was originally.