We’re load testing Openfire in preparation for a release of our new software and seem to be encountering a memory leak. We ramp up to about 350 users and start seeing the memory rise and rise and rise. Even when the users leave the memory usage keeps growing until eventually you get an out of memory error.
We’re using the latest Openfire, it’s a vanilla install with all plugins removed and we have tweaked the JVM settings. I noticed the PDF of Openfire scalability performance only shows about 5 minutes of testing under load - presumably this test was carried on for longer than that?
We’ve changed the tests so the users don’t even chat anymore, they simply join a room, wait, then leave. Memory use rises gradually then suddenly starts heading sky high (happy to provide the graphs). It does this time garbage collect after a certain amount of time.
I’m presuming it’s something we’re doing as I’m guessing there are people on here using Openfire in a productive environment under high load? The company I work for does online gaming and we’ve seen activity hit 1000 users in a room before. Right now we can’t prove we can get much beyond 250…
Yes, we have a script that generates all the SQL necessary to insert users. We then just pump that SQL into the database (e.g. msql < generatedUsers.sql -D openfire).
Thanks Guus, yes it does. We tried Tsung and got the same issues.
As an update, we’ve actually been running a load test now for about 17 hours, with 500 connected users talking at roughly 2 chats per second in a single chat room. Rather than use Smack we’re actually sending raw XML for room joins, messages etc. The reason we’re doing this is to mimic as closely as possible what it sent by XIFF (as our clients are all XIFF).
I think in our original tests we were putting an unrealistic load through the server which may have caused the problems. Our plan now is to start small and build up slowly - next step to increase the number of chat rooms in the load test, then do the same thing with all our plugins enabled.
Michael: just a warning: combining heavy MUC usage with the Openfire clustering feature (which Jive Software used to sell) can (will?) add unacceptable amounts of load to your environment. If you rely on MUC and use a clustered setup, be very cautious.
Thanks again Guss, is that from personal experience? I’ll pass that on to the rest of the team as we have been asked about clustering for the future. Have you had any experience (positive or negative) with the connection manager module?
Yes, that’s from personal experience. I used to work on a IM domain (Openfire based, although heavily modified) that had a user base in the 7-digit range. And yes, I have experience with connection managers. They are useful, to a certain extent. If your bottleneck is (client) socket handling (SSL, compression, they can help you out.