Too many open files on 3.4.2 Admin Console? (java.io.FileNotFoundException)

Openfire Openfire Support
1

Hi all,

We recently upgraded a production installation to 3.4.2 and just started seeing these 500 errors from Jetty when trying to use the Admin console. Nothing else has changed besides the version. Same config, memory settings, database connection, etc. (Linux Kernel 2.6, Sun JVM 1.6.0_01, Postgresql 8.2). The server is handling around 1,000 concurrent sessions and has a couple of hundred MUC rooms where about 50% are pretty active. It’s using only about 55MB of heap (of 750MB available to the JVM). The previous version that was in production (3.3.3) was handling the same load and never had this error.

Anybody else seeing this when trying to access the admin console? We’ve seen it in Safari and Firefox but it seems to happen more often in Safari. We’re tunneling HTTP through SSH. In this case it happened for index.html but it appears to happen randomly for any one or more files served by Jetty (css, images, etc.)

HTTP ERROR: 500

/usr/local/share/openfire-3.4.2/plugins/admin/webapp/index.html (Too many open files)

RequestURI=/

Caused by:

java.io.FileNotFoundException: /usr/local/share/openfire-3.4.2/plugins/admin/webapp/index.html (Too many open files)

at java.io.FileInputStream.open(Native Method)

at java.io.FileInputStream.(FileInputStream.java:106)

at org.mortbay.resource.FileResource.getInputStream(FileResource.java:281)

at org.mortbay.resource.Resource.writeTo(Resource.java:486)

at org.mortbay.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:662)

at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:416)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)

at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)

at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)

at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:268)

at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)

at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:465)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1093)

at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:11 8)

at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1084)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)

at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)

at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)

at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollect ion.java:211)

at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)

at org.mortbay.jetty.Server.handle(Server.java:313)

at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:506)

at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.j ava:830)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:396)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Powered by Jetty://

2

Hey ikester,

Which plugins are you using? We found a leak in the latest gateway plugin version when using buddy avatars (I think it was with MSN) that was generating the exact error you are seeing. I’m not a linux expert but I know that you can list the files being opened by a user/process. We used that information to figure out that the gateway was not releasing some files.

Regards,

– Gato

3

Hi Gato,

Thanks for the reply. We’re not using the gateway plugin but I’ll be sure to investigate open files next time this happens. We’re also going to update to 3.4.3 soon so we’ll see if it we get it again.

Ike

4

Hey there,

Try checking your ulimit setting when you get this error. If you don’t want to wait try setting the ulimit to unlimited.

Thanks,

Jay

5

Thanks. The open file descriptor limit for the shell that started the process was 1024 and we’re reaching that number just in user sessions so I guess that was the cause of it. Checking the open file descriptors for the JVM running openfire we clearly see one entry for every socket connected, so we need the limit to be at least the number of users expected plus a few dozen for all the JAR files, log files, etc. We’ll restart the service after doing ‘ulimit -n 65535’ and see if it happens again.

I just wonder how you guys tuned your systems when you profiled the server to up to 30K concurrent users per server. Any tips would be helpful.

Thanks again,

Ike