Openfire freezing after deploying Spark

Weird issue. Openfire has been running fine with 200 users using Exodus (old unsupported client). On Friday i started deploying latest nightly build of Spark (we were using Spark in testing group, about 10-15 users). After i have deployed it to 20-30 users Openfire freezed and i had to reboot the server. After two days another freeze. The only change is the Spark deployment. So it looks like Spark, a client, can freeze the server… Not a pretty start of migrating to Spark. I can’t pin point the issue yet. Also logs are empty and it seems they stopped working a few years ago (i think after migrating to log4j). I had issues before with JVM taking all the memory and Openfire freezing, so i have increased JVM memory and restart Openfire every month, and that was enough. Maybe that’s the same reason, though i’ve been checking JVM a few times per day and it was fine. Probably will need to incease VM memory and give more to JVM.

Openfire is fine to administer, but all those memory issues and the fact that a client can bring it down is frustrating… We are closer and closer to moving to Microsoft Lync with every such issue.

empty logs related stderror.log

log4j:ERROR setFile(null,true) call failed.

java.io.FileNotFoundException: /logs/debug.log (No such file or directory)

at java.io.FileOutputStream.open(Native Method)

at java.io.FileOutputStream.(FileOutputStream.java:212)

at java.io.FileOutputStream.(FileOutputStream.java:136)

at org.apache.log4j.FileAppender.setFile(FileAppender.java:289)

at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:167)

at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:163)

at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:256)

at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:220)

at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:15 0)

at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.ja va:163)

at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurat or.java:425)

at org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:394)

at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:829)

at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:712)

at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:618)

at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.jav a:470)

at org.apache.log4j.LogManager.(LogManager.java:122)

at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)

at org.jivesoftware.openfire.XMPPServer.(XMPPServer.java:150)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessor Impl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor AccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:525)

at java.lang.Class.newInstance0(Class.java:372)

at java.lang.Class.newInstance(Class.java:325)

at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:113)

at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:58)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja va:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)

at com.install4j.runtime.Launcher.main(Unknown Source)

log4j:ERROR setFile(null,true) call failed.

java.io.FileNotFoundException: /logs/info.log (No such file or directory)

at java.io.FileOutputStream.open(Native Method)

at java.io.FileOutputStream.(FileOutputStream.java:212)

at java.io.FileOutputStream.(FileOutputStream.java:136)

at org.apache.log4j.FileAppender.setFile(FileAppender.java:289)

at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:167)

at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:163)

at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:256)

at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:220)

at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:15 0)

at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.ja va:163)

at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurat or.java:425)

at org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:394)

at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:829)

at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:712)

at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:618)

at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.jav a:470)

at org.apache.log4j.LogManager.(LogManager.java:122)

at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)

at org.jivesoftware.openfire.XMPPServer.(XMPPServer.java:150)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessor Impl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor AccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:525)

at java.lang.Class.newInstance0(Class.java:372)

at java.lang.Class.newInstance(Class.java:325)

at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:113)

at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:58)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja va:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)

at com.install4j.runtime.Launcher.main(Unknown Source)

log4j:ERROR setFile(null,true) call failed.

java.io.FileNotFoundException: /logs/warn.log (No such file or directory)

at java.io.FileOutputStream.open(Native Method)

at java.io.FileOutputStream.(FileOutputStream.java:212)

at java.io.FileOutputStream.(FileOutputStream.java:136)

at org.apache.log4j.FileAppender.setFile(FileAppender.java:289)

at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:167)

at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:163)

at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:256)

at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:220)

at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:15 0)

at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.ja va:163)

at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurat or.java:425)

at org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:394)

at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:829)

at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:712)

at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:618)

at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.jav a:470)

at org.apache.log4j.LogManager.(LogManager.java:122)

at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)

at org.jivesoftware.openfire.XMPPServer.(XMPPServer.java:150)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessor Impl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor AccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:525)

at java.lang.Class.newInstance0(Class.java:372)

at java.lang.Class.newInstance(Class.java:325)

at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:113)

at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:58)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja va:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)

at com.install4j.runtime.Launcher.main(Unknown Source)

log4j:ERROR setFile(null,true) call failed.

java.io.FileNotFoundException: /logs/error.log (No such file or directory)

at java.io.FileOutputStream.open(Native Method)

at java.io.FileOutputStream.(FileOutputStream.java:212)

at java.io.FileOutputStream.(FileOutputStream.java:136)

at org.apache.log4j.FileAppender.setFile(FileAppender.java:289)

at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:167)

at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:163)

at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:256)

at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:220)

at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:15 0)

at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.ja va:163)

at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurat or.java:425)

at org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:394)

at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:829)

at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:712)

at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:618)

at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.jav a:470)

at org.apache.log4j.LogManager.(LogManager.java:122)

at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)

at org.jivesoftware.openfire.XMPPServer.(XMPPServer.java:150)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessor Impl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor AccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:525)

at java.lang.Class.newInstance0(Class.java:372)

at java.lang.Class.newInstance(Class.java:325)

at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:113)

at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:58)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja va:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)

at com.install4j.runtime.Launcher.main(Unknown Source)

I’ve been using good ol’ 2.6.3 for years. Last time i tried a beta/nightly build i had to reinstall stable after a few days. So my personal experience is that i wouldn’t recommend a nightly for production.

For you log4j problem you might need to check if the openfire daemon has writing permissions for OpenfireDir/logs folder. Or even if the folder exist. If it doesn’t, you should create it.

Also there’s a file named log4j.xml in OpenfireDir/lib that contains the configuracion for log4j.

A default config should be like this

<?xml version="1.0" encoding="UTF-8" ?>

<log4j:configuration xmlns:log4j=“http://jakarta.apache.org/log4j/”>

</log4j:configuration>

Daemon is running with jive:users user and logs folder and all files in it are owned bu jive:users. My log4j.xml is almost exactly the same, but after the error-out part it contains this:

<!-- OF-506: Jetty INFO messages are generally not useful. Ignore them $ and then the root part. I've been using every Spark nightly myself for a long time. Yes, there are some bugs or regressions occasionally, but the fixes included overweight this for me. Only the fix for Java6 bug with focus capturing from other active windows is already worth it. Even if this is some issue with Spark nightly, it shouldn't be that a client could bring down a server. Sending huge file doesn't affect a server, well it is set to be peer to peer, so it shouldn't. The only thing i can think of is maybe some weird image set as an avatar and making Openfire crazy. I think i remember something similar related to Smack or Openfire itself in the past.