Openfire appears to runaway after a period of days/weeks

Greetings I am currently running version 3.6.3 of openfire and overall I am loving it. Very stable and easy to administer. However I havea a strangeness that occurs after Openfire has run for a few days or weeks. It seems to spin up and start using A LOT of CPU. I run it on a Zlinux REDHAT IFL on a z9 machine (mainframe) The prior version 3.6.0a ran for months and months with no issues at all. This last weekend it spun up and ate up the whole IFL. I just wondered if anyone had had similar issues with Openfire apparently running away with itself. Please let me know if anyone has any ideas or thoughts or heard any good rumors.

Hi,

Could you tell us more about what plugins you use? Which backend database? Are the log files saying anything? Are you manually clearing cache, which can sometimes cause issues?

daryl

I use the client control, monitoring service and search plugins. My database is MYSQL. I do not manually clear cache. I am getting a ton of these warnings at the time ;

2009.04.18 12:32:34 Closing session due to incorrect hostname in stream header. Host: jabber.llbean.com. Connection: org.jivesoftware.openfire.net.SocketConnection@35c435c4 socket: Socket[addr=/172.16.252.7,port=32824,localport=5269] session: null
2009.04.18 12:37:51 Closing session due to incorrect hostname in stream header. Host: jabber.llbean.com. Connection: org.jivesoftware.openfire.net.SocketConnection@10391039 socket: Socket[addr=/172.16.252.7,port=32842,localport=5269] session: null
2009.04.18 12:43:03 Closing session due to incorrect hostname in stream header. Host: jabber.llbean.com. Connection: org.jivesoftware.openfire.net.SocketConnection@7e527e52 socket: Socket[addr=/172.16.252.7,port=32854,localport=5269] session: null
2009.04.18 12:48:08 Closing session due to incorrect hostname in stream header. Host: jabber.llbean.com. Connection: org.jivesoftware.openfire.net.SocketConnection@150d150d socket: Socket[addr=/172.16.252.7,port=32861,localport=5269] session: null
2009.04.18 12:54:21 Closing session due to incorrect hostname in stream header. Host: jabber.llbean.com. Connection: org.jivesoftware.openfire.net.SocketConnection@10791079 socket: Socket[addr=/172.16.252.7,port=32872,localport=5269] session: null
2009.04.18 12:59:30 Closing session due to incorrect hostname in stream header. Host: jabber.llbean.com. Connection: org.jivesoftware.openfire.net.SocketConnection@29e629e6 socket: Socket[addr=/172.16.252.7,port=32879,localport=5269] session: null
2009.04.18 13:04:43 Closing session due to incorrect

I see these type of errors in the error log. However I don’t know if these are ‘normal’ types of errors.

2009.04.18 03:10:58 [org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngi ne.java:373)] Error sampling for statistic packet_count
org.jrobin.core.RrdException: Bad sample timestamp 1240038600. Last update time was 1240038600, at least one second step is required
at java.lang.Throwable.(Throwable.java:57)
at java.lang.Throwable.(Throwable.java:68)
at org.jrobin.core.RrdException.(Unknown Source)
at org.jrobin.core.RrdDb.store(Unknown Source)
at org.jrobin.core.Sample.update(Unknown Source)
at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngin e.java:367)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:432)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 665)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
at java.lang.Thread.run(Thread.java:797)
2009.04.18 03:10:58 [org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngi ne.java:373)] Error sampling for statistic muc_incoming
org.jrobin.core.RrdException: Bad sample timestamp 1240038600. Last update time was 1240038600, at least one second step is required
at java.lang.Throwable.(Throwable.java:57)
at java.lang.Throwable.(Throwable.java:68)
at org.jrobin.core.RrdException.(Unknown Source)
at org.jrobin.core.RrdDb.store(Unknown Source)
at org.jrobin.core.Sample.update(Unknown Source)
at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngin e.java:367)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:432)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284)

Just a ping to bump this guy? Anyone? Hello Hello?

Hi Septimusx,

I saw this error to on my new installation of openfire 3.6.3:

Closing session due to incorrect hostname in stream header …

Since I only have one server, so I disabled the Server-to-Server service through Server Setting in Admin consolde. This stop the error message.

These past few days my openfire becomes unstable. For some unknown reason it stop connecting to the AD intermittenly which cause nobody can login to the jabber and admin console. I have to restart the openfire service. Did you experience the same issue? I have posted my issue in this forum.

But still nobody comes up with the solution.

Cheers,

Charlie

Nope, Sorry Charlie. I haven’t ever had that particular issue occur. It just seems that after a few weeks or even a month or two, the openfire server start to wind up and eat the linux that it is running on. Bouncing the server fixes the issue (temporarily) but I was hoping that someone might have a real fix. The lack of an answer tells me I might be waiting until the next release.

I am not sure why you think there will be another release for there is currently nobody developing openfire and commiting code. You may wish to look elsewhere.

daryl

Thanks for your answer. Why wouldn’t I think there would be another release? How would I know no one is developing more code? I am not a developer of this project I am a user. I come to this forum to ask questions and hopefully get answers from helpful users or developers. More often than not my questions usually go unanswered, or are answered with things that are less than helpful. Charlie provided info that could prove to be useful as I persue this issue. Would you perhaps have something actually constructive to add?

If you have been reading the forums, you probably have seen the huge security hole in the current release of Openfire. Are you running the binary I provided to the community to patch this hole?

daryl

No I am not terribly concerned about it since I do not allow the Openfire service to have access to the outside. I realize that that is a poor attitude, however I am relying on my security folks to be my first line of defense. That being said I did see the patch and have the intention of putting it on. Are you of the opinion that this hole could cause the process to runaway? If so I would make the patch a much higher priority. Thank you.

Is the openfire processing eating all CPU or memory or both?

What clients are your users using? Do you have any bots or other exotic things that could cause openfire to start thrashing packets internally? Maybe try installing the stats plugin and see if it shows any suspiciously large activity.

daryl

Ooh good idea on the stats. I will do that definitely. It chews cpu I believe not memory though I will double check and make sure on that. I allow two clients, Pandion and Exodus. I have no other bots or funky things.

Hey @Septimusx,

Next time you have a constant CPU spike (not sometihg intermitent) you might want to get a thread dump of the JVM (kill -3 [process id]) and paste it here. That information should help figure out what is going on inside of the server.

BTW, a new version of the server has been released today that includes the fix to the security hole.

Tks,

– Gato

Hi Gato,

Sorry for being out of the topic.

BTW … Do you have any ideas what might cause the issue on my newly installed openfire 3.6.3 as I mentioned in other thread:

I saw a new version has been released. Do you think it will solve my issue if I upgrade mine?

Thanks in advance.

regards,

Charlie

Hey Charlie,

Haven’t seen that other thread but if you are also seeing high CPU consumption then getting a thread dump is something that will help get more information about what is going on. On the other hand, if you see that the allocated memory is 100% used then garbage collection is the reason for the high CPU spike. In this latter case, a thread dump will probably not help and instead a heap dump is what you need to get so we can see what is consuming so much memory. If there is just no memory leak but you are still consuming all the memory then in this case you just need to increase the memory allocated for the server.

The new version will not fix the problem you are solving.

Regards,

– Gato

Thanks Gato. I will definitely attempt to get a JVM dump next time it happens. I appreciate your info.

Hi all,

I got the same problem, after Openfire3.6.3 has run for a 5 or 7 days then it was shutdown, I don’t know the reason why.

Anyone help us for this issue.

Thanks,

Alextour

Just to make sure that your problem is indeed the same. Openfire appeared to spike and use up the machine it was on. Then we purposely shut it (the server) down and restarted the server. After that openfire started and ran normally. Since posting this issue it has yet to happen again.