We are running Openfire 4.8.3 in a cluster. After a few days one of the nodes stops accepting connections. When trying to access the Admin UI of that node we are faced with a 500 error code.
On the logs we see the following exception.
ERROR [PluginMonitorTask-2]: org.jivesoftware.openfire.container.PluginMonitor - An unexpected exception occurred:
java.nio.file.FileSystemException: /opt/openfire/plugins: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:440) ~[?:?]
at java.nio.file.Files.newDirectoryStream(Files.java:614) ~[?:?]
at org.jivesoftware.openfire.container.PluginMonitor$MonitorTask.run(PluginMonitor.java:223) [xmppserver-4.8.3.jar:4.8.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
This issue goes away with a reboot of the service and reappears after a few days.
We have very low load (5-10 users).
This “too many open files” is probably the culprit for the 5XX codes in the admin UI since the system cannot open the file “/opt/openfire/plugins/admin/webapp/WEB-INF/sitemesh.xml”
Linux usually establishes a limit of 1024 open file descriptors per process/user. You should increase it via something like the following, which raises to 64k
Thank you for the suggestion.
It is a bit strange that this happens to the same node with this frequency. The requests forwarded by my load balancers (HAPROXY) are not more than the other nodes, actually the opposite is true.
Isn’t there anything in the above error that would point to the actual issue?
As stated we are looking at very very low load. We have less than 10 users.
I should note however we are checking the nodes to see if any is down with the load balancer.
Could it be that the load balancer is generating many requests to the nodes that result in exhaustion in open files?
Have you run into a similar issue with load balancing in the past?
On further investigation we are seeing the following exceptions on this node. I hope this will help.
2024.09.05 10:56:45.613 ERROR [PluginMonitorExec-3]: org.jivesoftware.util.XMLProperties - Error reading XML properties
org.dom4j.DocumentException: Error on line 1 of document : Premature end of file.
at org.dom4j.io.SAXReader.read(SAXReader.java:511) ~[dom4j-2.1.4.jar:?]
at org.dom4j.io.SAXReader.read(SAXReader.java:408) ~[dom4j-2.1.4.jar:?]
at org.jivesoftware.util.XMLProperties.buildDoc(XMLProperties.java:810) [xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.util.XMLProperties.(XMLProperties.java:160) [xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.openfire.index.LuceneIndexer.loadPropertiesFile(LuceneIndexer.java:459) [monitoring-2.5.0.jar:?]
at org.jivesoftware.openfire.index.LuceneIndexer.start(LuceneIndexer.java:80) [monitoring-2.5.0.jar:?]
at org.jivesoftware.openfire.plugin.MonitoringPlugin.initializePlugin(MonitoringPlugin.java:206) [monitoring-2.5.0.jar:?]
at org.jivesoftware.openfire.container.PluginManager.loadPlugin(PluginManager.java:652) [xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.openfire.container.PluginMonitor$MonitorTask$4.call(PluginMonitor.java:380) [xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.openfire.container.PluginMonitor$MonitorTask$4.call(PluginMonitor.java:368) [xmppserver-4.8.3.jar:4.8.3]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204) ~[?:?]
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178) ~[?:?]
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400) ~[?:?]
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327) ~[?:?]
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1465) ~[?:?]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1013) ~[?:?]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605) ~[?:?]
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112) ~[?:?]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:542) ~[?:?]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:889) ~[?:?]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:825) ~[?:?]
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[?:?]
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224) ~[?:?]
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:637) ~[?:?]
at org.dom4j.io.SAXReader.read(SAXReader.java:494) ~[dom4j-2.1.4.jar:?]
… 13 more
2024.09.05 10:56:45.618 ERROR [PluginMonitorExec-3]: org.jivesoftware.openfire.archive.ArchiveIndexer[CONVERSATION] - An exception occurred while initializing the Lucene index that is expected to exist in: /opt/openfire/monitoring/search
java.io.IOException: Error on line 1 of document : Premature end of file.
at org.jivesoftware.util.XMLProperties.buildDoc(XMLProperties.java:814) ~[xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.util.XMLProperties.(XMLProperties.java:160) ~[xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.openfire.index.LuceneIndexer.loadPropertiesFile(LuceneIndexer.java:459) ~[monitoring-2.5.0.jar:?]
at org.jivesoftware.openfire.index.LuceneIndexer.start(LuceneIndexer.java:80) [monitoring-2.5.0.jar:?]
at org.jivesoftware.openfire.plugin.MonitoringPlugin.initializePlugin(MonitoringPlugin.java:206) [monitoring-2.5.0.jar:?]
at org.jivesoftware.openfire.container.PluginManager.loadPlugin(PluginManager.java:652) [xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.openfire.container.PluginMonitor$MonitorTask$4.call(PluginMonitor.java:380) [xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.openfire.container.PluginMonitor$MonitorTask$4.call(PluginMonitor.java:368) [xmppserver-4.8.3.jar:4.8.3]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
2024.09.05 10:56:45.619 ERROR [PluginMonitorExec-3]: org.jivesoftware.openfire.container.PluginManager - An exception occurred while loading plugin ‘monitoring’:
java.lang.NullPointerException: Cannot invoke “org.jivesoftware.util.XMLProperties.getProperty(String)” because “this.indexProperties” is null
at org.jivesoftware.openfire.index.LuceneIndexer.getLastModified(LuceneIndexer.java:187) ~[monitoring-2.5.0.jar:?]
at org.jivesoftware.openfire.index.LuceneIndexer.start(LuceneIndexer.java:154) ~[monitoring-2.5.0.jar:?]
at org.jivesoftware.openfire.plugin.MonitoringPlugin.initializePlugin(MonitoringPlugin.java:206) ~[monitoring-2.5.0.jar:?]
at org.jivesoftware.openfire.container.PluginManager.loadPlugin(PluginManager.java:652) [xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.openfire.container.PluginMonitor$MonitorTask$4.call(PluginMonitor.java:380) [xmppserver-4.8.3.jar:4.8.3]
at org.jivesoftware.openfire.container.PluginMonitor$MonitorTask$4.call(PluginMonitor.java:368) [xmppserver-4.8.3.jar:4.8.3]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]