Openfire 4.3.2 Performance issue [Waiting for lock]

Hi all,
We are using Openfire-4.3.2 and we are facing performance issue.It seems threads are waiting for some kind of lock after counts of users hits 5000 or so.The attachment is the java heap dump.
Any help will be appreciated.

what OS are you running? My guess is your running openfire on linux. if so, you may wan to check this out
https://www.tecmint.com/increase-set-open-file-limits-in-linux/

Yes linux but it’s not the open file limit or something like that.The resource limits are basically unlimited.
Would you mind looking at the attachment I added?

ahh…I think this might be a mina bug.I think this has been fixed.

That thread dump was very helpful, thanks for including that. It shows that a lot of threads are waiting for the same thing: a lock on the User cache. When the thread dump was being created, the cache was being culled.

This is a strong indication that the capacity of this cache is to small for your use-case.

Try increasing the cache size, by adding these system properties (or increase their value, if they’re already defined on your server:

  • cache.userCache.size - maximum cache capacity, in size. Default value: 524288
  • cache.username2roster.maxLifetime - maximum time a cache entry can be served from cache, until it needs to be refreshed, in milliseconds. Default value: 1800000

Note that you’ll likely need to restart Openfire for the configuration changes to take effect.

It’s possible that other caches suffer from similar problems. I advice you to take a look at the “cache summary” page in the admin console, which lists usage statistics for every cache:

1 Like

Changed cache settings but it didn’t work.Once the number of sessions reached about 6000, Openfire stopped creating new sessions meanwhile destroying connected ones.
Here is another thread dump.jstack.txt (189.1 KB)

In your last thread dump, I’m no longer seeing resource contention around that cache. The change at least had some effect. I’m sad to hear it’s not resolving the larger issue that you have.

I’m unsure how to proceed next. Are there any relevant messages in log files?

Not so much.It just printed a lot of destroying session…

any chance you would mind spinning this up on a windows host? Id be curious to see if you hit the same limited with another OS. It might offer up some more clues.

Sorry we don’t run any windows host…Is there a limit for the maximum counts of logins of a single user? We actually have a few users but each user may have 10k or so connections which login with different resource name.

Do the users have any roster at all?
Is stream management disabled?
Are you setting any JVM garbage collection special flags?
When openfire refuses new connections, does the admin console indicate there is memory left?
Your company should hire Guus out for a day, I am sure he would get to the bottom of it :slight_smile:

1 Like