Problems with 2000 concurrent users

I have experienced problems with my openfire chat server. Yesterday I had about 1900 concurrent users chatting in 5 different rooms. The room with the most users had about 1000 users.

The different rooms are football chat rooms, and in the end of the game people wrote alot of messages due to a goal and this caused serious problems for my openfire server. The delay was crazy. When writing a message users would have to wait minutes before it was displayed.

Until this happend the chat went well.

I use a load balancer (round robin) towards two connection managers.

My openfire server is a Intel Xenon 3.00GHz (dual core) with 8GB of ram

After these problems I went in and saw that the memmory usage of the server was 5.7 GB

My 2 connection managers are both Intel Xenon 3.00GHz (dual core) with 4GB of ram

So if I have 1000 users in one room and 50 of them go crazy when there is a goal and send 2 messages each in a 10 second period this accumulates (5021000)=100.000 messages over a 10 second period.

So my question is does anyone now what my problem might be? Is it the 5.7GB og used RAM that is causing the problem? Do I simply need more RAM? I have disabled logging in the rooms.

Hope anyone have some ideas of what might be the problem. About 10 minutes after the crazy delay the chat worked normally again (I did not need a restart. After about 4 hours later the memory usage was at about 5,2 GB (out of 8).

After having restarted openfire the memory usage was 1.55 GB.

In openfire.vmoptions I have -Xms1024m anf -Xmx1024

Thanks

Hi,

where did you see the 5.7 GB memory usage? As long as you set “-Xmx1024m” Openfire will not use more than 1 GB within the JVM. Ant the JVM process will likely use less than 1.4 GB.

LG

Well I saw in windows task manager. Here under performance the Memory usage was 5.7GB out of 8 GB. When I restart openfire it drops down to 1.55GB so I figured it has to openfire that are taking the memory.

I realize that all this isnt openfire. At the moment my total memory usage in under performance is 2.24GB while the java.exe process takes 1.2GB

And sorry I saw that Im starting openfire from a bat file where I set the following parameters.

-Xms 4069m -Xmx 6144m

Should I increase the -Xmx value? (I guess I should be careful with this running at almost max now)?

Hi,

you my want to enable a “gc.log” for Openfire. So you see when the JVM does perform a garbage collection and how much memory is free after a gc. This should help you to understand whether Openfire was busy with garbage collections - then it may make sense to increase the Xmx value - or whether it was a problem with the traffic.

JVM Settings and Debugging

LG

So Im not sure if Im right but do I add

java -server -XX:+PrintGCDetails -Xloggc:/tmp/gc.log" …

to get the gc log? Or do I need install4J? I saw this cost money.

And do I need the -Xmx value? I guess the prossess will get the available memory anyway? And then it would be better not to set it to use all available memory?

Hi,

as you use a bat file you are likely on Windows. So you may want to add to the openfire.vmoptions file:

-XX:+PrintGCDetails

-Xloggc:c:/temp/gc.log

Anyhow Openfire must be able to write to c:/temp, you could also use c:/progra~1/openfire/logs/gc.log.

You need to set the Xmx value and it must be below the physical server memory. If you don’t set it the JVM will likely not allocate so much memory.

LG

Ok so now Ive added this… So ill wait until after the weekend to see what happends.

But when I set

-Xms 4069m -Xmx 6144m

and restart openfire in window task manager the memory usage is only 1.55GB and the java.exe process takes 450000k

So its not as I thought that it would take 4069m at startup?

I also installed the java-monitor plugin. Hopefully that can also tell me something.

Thanks for the help

Hi,

the task manager may display something wrong, this is common for Windows. In the gc.log you should see that the process uses 4 GB as soon as a garbage collection does occur.

LG