Leaking file handles?

Sometimes our instance of openfire will start spewing errors like the following:

2008.05.31 18:22:36 EXCEPTION

java.io.IOException: Too many open files

at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)

at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConne ctor.java:75)

at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:475)

at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:166)

at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java :124)

at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:537)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450)

Once we see this error, the server typically needs to be restarted. Our open file limit is set much higher than our concurrent user count (15,000 users, but the open file limit is set at 128k) This seems to happen when our servers are lagging and many people are trying to reconnect.

Is it possible that OF is leaking file handles somehow? Or is it just that the sockets stick around for several minutes after being closed, and we need to set the file limit much higher? Any advice on this would be greatly appreciated.

What version of the IM Gateway plugin do you have installed? (or do you have it installed?) There was a bug with it before where MSN avatar support was causing this, but it was resolved.

We don’t have the IM gateway plugin installed. We are going to increase the file handle limit by a large amount and see if the problem just goes away. If it doesn’t, I suppose that would indicate a leak somewhere? Any ideas on tools we can use to try and debug this issue?

Edit: We’re running on CentOS, if that matters

I’m also running on CentOS for what it’s worth.

Lets see… Well, maybe provide a list of all of the plugins you do have installed?

We are running a pretty minimal set of plugins, just MoTD and the Content filter. We also have a custom plugin which handles some internal logic, although we are in the process of moving that to an external component. I am going to try and take some time today playing around with the lsof utility to see if I can determine what is going on with the open files. I’ll report anything I find here!


an “lsof” output would be great so one could see where the leak occurs / which “files” are open.