We got into OOME issue in our production environemnt.
The OOME issue is happening on the daily basis. We have setup JVM heap size to 1024 MB , i.e. -Xmx1024. We are running with about 1000+ concurrent users. The functioanlyty that openfire serves in our project is like -
- After every 1 min each user sends some information in XML format . Along with this XML data it also establishing its prescnce again.
All this user sends their packet to a single chatroom.
-
So after every min openfire will get 1000+ packets simultanesoly from all users.
-
We have a single consumer which is listening this chatroom and parsing the incoming packets.
-
Also our user’s will always be connected to openfire , they never logout.
-
This is casuing the openfire server to went OOME every day.
-
We have a plan to run around 3000+ users in next week.
We analyzed the heap dump and its says like **writeRequestQueue inside MINA **is failing.
Also some internal cache queue of openfire is failing inside some cacheWrapper class.
TO fix this issue we have given following tries -
-
Remove unused plugins including monitoring plugin , user search plugin.
-
set the following property - cache.username2roster.maxlifetime=86400000 (1 day)
cache.username2roster.size=33554432 (32 MB)
- xmpp pep is already false
All this measures doesnt give us success. We are still getting OOME.
We have some question as well -
Question 1 - Can we directly compile the latest mina code in openfire, i,e, replace the existing code of mina in openfire.jar?
Question2 - Are their any properties that we are missing to set for our specific requuiremment?
Question3 - What is the best garbage collector settings?
Question4 - Are their any other cache setting need to add?
Please suggest some suitable solution for that. Any help will be greatly appreciated.
Waiting for your quick response.
Thanks,
Keshav