Increased memory usage

D_P · June 21, 2007, 4:17am

I’‘m actually not sure if this is a bug or not. However, last night after installing 1.1.0 beta 2, everything was working fine. When I woke up this morning, the server was dead and giving me a few errors about the Java heap being full. So, I restarted the server and everything was fine. I’‘ve been operating my server with the default memory allocation for a long time now, but thought that maybe this new functionality in the IM plugin would require additional memory. So I upped it to 256MB and 512MB for minimum and maximum heap sizes. I think the default memory allocation is 25MB? Something like that. I can’'t remember off the top of my head.

Anyway, when I logged into the admin console tonight, I saw that my memory usage is up to about 110MB, though I don’‘t have any more users on than I did before. Is it possible there’‘s some sort of memory leak in the gateway, or is it normal for OpenFire’‘s memory usage to increase just because you give it more memory to work with? I’‘ve not fiddled with that before, so I don’‘t know what to expect. But I thought I’'d comment here anyway.

MysticOne

Daniel_Henninger · June 21, 2007, 11:20am

Well, I mean, you -are- probably running 8 different transports and a host of extra libraries are loaded. It wouldn’‘t surprise me to see the memory usage increase a bunch. However if it keeps steadily increasing then yes there’‘s probably something that’'s not being cleaned up properly.

D_P · June 21, 2007, 1:04pm

Yeah, that’'s what I was thinking. Since there needs to be a new connection for each user using each gateway, then all the stuff that goes along with that, the memory usage should increase. Are all the increases confined to the Java VM? So if I need more memory, increasing the min/max heap amount should do the trick?

I checked this morning and it was pretty constant. I think 112MB for 11 users. It was a little lower last night with 10 users, but not by much, so it seems consistent thus far.

MysticOne

Daniel_Henninger · June 21, 2007, 1:40pm

In theory it shouldn’‘t keep increasing -that- much. At some point it’'ll have everything it needs loaded into memory across the board and should only have minor increases with different numbers of users. … I think. =D I am not uber when it comes to knowing how java handles memory. hehe

Increasing the max heap should do it though.

D_P · June 22, 2007, 12:41am

Uh oh. I got home from work today and looked at it, and with 14 users, it’‘s up to 233.3MB out of 256MB. Before this, I never had more than 25MB or so (again, whatever the default is, I can’‘t remember) allocated. Out of the 14 users I have online, 5 are signed into accounts via the gateway. Those 5 have 15 active accounts on the gateway altogether. 5 AIM, 5 MSN, 2 ICQ, 1 IRC and 2 Yahoo. I don’‘t know if that’‘s helpful or not, so if there’'s anything else you can think of that would let you know if the memory leak is related to the gateway (and if so, where it is) just let me know.

MysticOne

Scott_Reynolds · June 22, 2007, 8:06pm

Default allocation on my system was ~128 MB. With everything connected and running for 3 unique JIDs I rarely see more than 30 MB in use. However, if I leave the gateway running for several days, there’'s an event that seems to trigger a rapid spiral. The last time it happened I estimated I had about 10 hours until I exhausted RAM. (This seems to happen with just the AIM and MSN transports loaded.)

My temporary solution on this server was to go back to 1.02. Unfortunately, I can’'t do too much experimentation here because my family depends on this being useful to some greater or lesser degree.

Daniel_Henninger · June 22, 2007, 9:05pm

Hrm. I’'ve been running it at home for a while now and:

39.77 MB of 63.31 MB (62.8%) used

I never see memory spikes. Wonder why y’‘all are seeing it? Either way, I’‘m not going to have time to get to most of this stuff soon most likely, so you may wish to downgrade to 1.0.2 if the beta is causing you problems. Of course that’‘s why it’'s a beta and not a release. =D

D_P · June 22, 2007, 9:27pm

Oh, I see how it is. You give us lovely new features, ask us to test them. Then when we find a bug, you’‘re too busy. But we’'re already addicted!

Yeah, it looks like it’‘s doing it again to me today. I’‘m going to downgrade when I get home, which sucks because I really have enjoyed the IRC functionality. Again, though, if there’‘s anything I can send to you that’‘d be helpful, let me know how/what and I’'ll get it to you. Thanks!

MysticOne

Scott_Reynolds · June 22, 2007, 9:52pm

jadestorm wrote:

I never see memory spikes. Wonder why y’'all are seeing it?

I’'ve gone 5 days without this happening but it seems to be inevitable.

Did I mention that I typically have 2-5 clients connected at the same time…? Perhaps this is related to the Concurrency issues thread?

Just a thought.

Daniel_Henninger · June 23, 2007, 1:13am

I don’'t suppose I could get you all to try out:

http://www.igniterealtime.org/forum/thread.jspa?messageID=149851&tstart=0#149851

Clearly noted memory issues =D I’'d actually be surprised if it really does effect the plugin memory issues, but it would at least eliminate (well mostly) known memory issues from the list of possible issues.

Daniel_Henninger · June 23, 2007, 3:30am

Also, some information to ask you all about …

Did you see this in 1.1.0 Beta (1)?
Are you suddenly using transports you weren’'t before because they are better or are new?

2a. If so, if you disable those transports does the problem vanish?

What’'s the latest version of Openfire that you were running where you saw the issues?
Since you installing the beta 2 plugin, have you restarted your server at least once?
Does your CPU load happen to spike as well?

Next time one if you see this happening, please find the process id of the java process running openfire (i say that to make sure you avoid the one that may exist that is a /bin/sh starter process) … and kill -QUIT it. (this won’‘t stop openfire, it will generate a thread dump to stdout (or maybe stderr, I forget) for the java process. I must admit I don’'t 100% know where that output is going to go, but look around your log files. Typically it goes to a nohup.out file. Email said file to me or send it via a private message?

Scott_Reynolds · June 23, 2007, 6:37pm

jadestorm wrote:

I don’'t suppose I could get you all to try out:

http://www.igniterealtime.org/forum/thread.jspa?messageID=149851&tstart=0#149851

Clearly noted memory issues =D I’'d actually be surprised if it really does effect the plugin memory issues, but it would at least eliminate (well mostly) known memory issues from the list of possible issues.

Already made that connection and updated.

jadestorm wrote:

Also, some information to ask you all about …

Did you see this in 1.1.0 Beta (1)?

Yes.

Are you suddenly using transports you weren’'t before because they are better or are new?

2a. If so, if you disable those transports does the problem vanish?

No – only AIM and MSN. I suspected Yahoo! and disabled it, but the problem returned.

What’'s the latest version of Openfire that you were running where you saw the issues?

I’'ve only ever tried the gateway with 3.3.1 (and, well, now 3.3.2 as well).

Since you installing the beta 2 plugin, have you restarted your server at least once?

Yes – uninstalling the gateway typically did not release all of the memory (though it did release a considerable amount). The last time this happened I was up to about 90 MB in use. Removing the gateway got me down to about 30 MB in use. This contrasts with about 20 MB in use without the gateway running, once memory usage has stabilized.

Does your CPU load happen to spike as well?

Yes, it does, though I’'m not sure of the degree.

Next time one if you see this happening generate a thread dump Email said file to me or send it via a private message?

Will do. I got brave or foolish (you pick which) and updated from svn trunk last night, before I saw your message. I’'m still using just AIM and MSN for now.

Daniel_Henninger · June 23, 2007, 11:17pm

scottr wrote:
jadestorm wrote:

I don’'t suppose I could get you all to try out:

http://www.igniterealtime.org/forum/thread.jspa?messageID=149851&tstart=0#149851

Clearly noted memory issues =D I’'d actually be surprised if it really does effect the plugin memory issues, but it would at least eliminate (well mostly) known memory issues from the list of possible issues.

Already made that connection and updated.

Ok =) Good to know!

jadestorm wrote:

Also, some information to ask you all about …

Did you see this in 1.1.0 Beta (1)?

Yes.

Are you suddenly using transports you weren’'t before because they are better or are new?

2a. If so, if you disable those transports does the problem vanish?

No – only AIM and MSN. I suspected Yahoo! and disabled it, but the problem returned.

Hrm. That’‘s quite troubling. I can only assume this has something to do with the new contacts manager and that I’‘m not properly cleaning it up. (that’‘s the biggest change I’‘ve made to the entire infrastructure) Still, I’‘ve got a lot of contacts, and it’‘s odd that I’'m not seeing this.

What’'s the latest version of Openfire that you were running where you saw the issues?

I’'ve only ever tried the gateway with 3.3.1 (and, well, now 3.3.2 as well).

Since you installing the beta 2 plugin, have you restarted your server at least once?

Yes – uninstalling the gateway typically did not release all of the memory (though it did release a considerable amount). The last time this happened I was up to about 90 MB in use. Removing the gateway got me down to about 30 MB in use. This contrasts with about 20 MB in use without the gateway running, once memory usage has stabilized.

Well I mean, it’‘s a large plugin, so it’'s bound to cause a bit of an increase. But in theory I would think it would stay relatively stable after an initial increase. What does it tend to be if you are running 1.0.2?

Does your CPU load happen to spike as well?

Yes, it does, though I’'m not sure of the degree.

Course if I’'m blowing memory out of the water somehow, I might also be causing swapping.

Next time one if you see this happening generate a thread dump Email said file to me or send it via a private message?

Will do. I got brave or foolish (you pick which) and updated from svn trunk last night, before I saw your message. I’'m still using just AIM and MSN for now.

Thanks! And as for trunk, I certainly don’‘t “condone” using trunk. ;D I often commit things that don’'t work yet. So “fair warning”.

D_P · June 23, 2007, 11:58pm

1. Did you see this in 1.1.0 Beta (1)?

Yes. Though, I didn’‘t know it at the time. This is what was causing my server to crash when I mentioned to you that it was crashing. But, it’'s the same type of behavior.

+2. Are you suddenly using transports you weren’'t before because they are better or are new?

2a. If so, if you disable those transports does the problem vanish?+

The only one I’‘m using that’‘s different is IRC. But it was always enabled before, I just wasn’'t using it as much. Other than that, I have AIM, ICQ, MSN, Yahoo and IRC all enabled.

3. What’'s the latest version of Openfire that you were running where you saw the issues?

3.3.1. I’'ve not yet had a chance to update to 3.3.2, but will try to do it later this evening.

4. Since you installing the beta 2 plugin, have you restarted your server at least once?

Yes, every time it crashes.

5. Does your CPU load happen to spike as well?

Not particularly. It does spike a little if I start swapping, but otherwise there’'s not much increased CPU activity that I can attribute to OpenFire.

In answer to your other question, I’‘ve had my server running all day with 1.0.2 (which I went back to this morning). I’‘m using 25MB out of 256MB, and it’‘s not gone much above that. For this amount of time with the beta of the plugin, I’'d be up to 250MB.

MysticOne

Klaas_Tammling · June 25, 2007, 8:36am

I’‘ve got the same problem now. The server is working fine for about 24 hours then when I wake up in the morning: server dead and memory full. CPU is idle but I can’‘t use the jabberd anymore. I’'m using Openfire 3.3.2 with AIM, ICQ, MSN, YIM, Google Talk and SIP enabled.

Darren_Sykes · June 25, 2007, 11:34am

It’'d be great to know which transport is causing this (if any).

I’‘d like to upgrade from my (stable) beta 1 to beta 2 because of the concurrency bug fixes but I don’'t want to hit the memory consumption issue.

Any clues? Just MSN and ICQ used here…

D

Klaas_Tammling · June 25, 2007, 11:45am

I don’'t know if it helps but these are the messages I had this morning when the server got overloaded.

2007.06.25 07:33:18 org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHandle r.java:109)