Good news (I think!) I’m not seeing the spinning process with the latest RC-1 RPMs and I was able to log into the admin console using my LDAP username and password.
I’ll keep a close eye on it and let you know if I see the spinning manifest itself
Darn spoke too soon, checked it this morning and there’s a 100% CPU spinning process:
31696 daemon
20 0 18.7g 5.8g 13m R 100.4 18.6 996:54.88 java
31607 daemon
20 0 18.7g 5.8g 13m S 0.3 18.6 0:32.58 java
31663 daemon
20 0 18.7g 5.8g 13m S 0.3 18.6 0:10.39 java
1 root
20 0 19356 1552 1252 S 0.0 0.0 0:00.60 init
Thread dump via kill -3 attached with Garbage Collection stuff stripped out
Installed the fixed DEB yesterday morning, everything is running smoothe since then.
Mark,
Please verify that the .jar files found in your $openfireHome/lib/ folder are circa Wednesday and not some earlier vintage. I am hoping that you are running a slightly older build.
Well as I’m using a Redhat derivative Openfire is installed via RPM so it’s highly unlikely but to confirm:
[root@tattle lib]# ls -alrt
total 26996
-rw-r–r-- 1 daemon daemon 9475616 Jun 8 16:12 openfire.jar.mina
-rw-r–r-- 1 daemon daemon 71510 Jun 10 21:21 startup.jar
-rw-r–r-- 1 daemon daemon 8870 Jun 10 21:21 slf4j-log4j12.jar
-rw-r–r-- 1 daemon daemon 95806 Jun 10 21:21 servlet-api.jar
-rw-r–r-- 1 daemon daemon 642809 Jun 10 21:21 postgres.jar
-rw-r–r-- 1 daemon daemon 10263042 Jun 10 21:21 openfire.jar
-rw-r–r-- 1 daemon daemon 87543 Jun 10 21:21 npn-boot.jar
-rw-r–r-- 1 daemon daemon 954041 Jun 10 21:21 mysql.jar
-rw-r–r-- 1 daemon daemon 362975 Jun 10 21:21 mail.jar
-rw-r–r-- 1 daemon daemon 2848 Jun 10 21:21 log4j.xml
-rw-r–r-- 1 daemon daemon 294726 Jun 10 21:21 jtds.jar
-rw-r–r-- 1 daemon daemon 188164 Jun 10 21:21 jetty-schemas.jar
-rw-r–r-- 1 daemon daemon 74639 Jun 10 21:21 jdic.jar
-rw-r–r-- 1 daemon daemon 36611 Jun 10 21:21 javax.websocket-api.jar
-rw-r–r-- 1 daemon daemon 77056 Jun 10 21:21 jasper-runtime.jar
-rw-r–r-- 1 daemon daemon 407502 Jun 10 21:21 jasper-compiler.jar
-rw-r–r-- 1 daemon daemon 641570 Jun 10 21:21 hsqldb.jar
-rw-r–r-- 1 daemon daemon 112341 Jun 10 21:21 commons-el.jar
-rw-r–r-- 1 daemon daemon 2842667 Jun 10 21:21 bcprov-jdk15on.jar
-rw-r–r-- 1 daemon daemon 605519 Jun 10 21:21 bcpkix-jdk15on.jar
-rw-r–r-- 1 daemon daemon 255692 Jun 10 21:21 bcpg-jdk15on.jar
drwxr-x— 10 daemon daemon 4096 Jun 11 13:15 …
drwxr-xr-x 2 daemon daemon 4096 Jun 11 13:15 .
As you can see everything bar “openfire.jar.mina” has the same timestamp. I can try removing that but as it doesn’t have a .jar extension I don’t think it should be loaded
Hey Mark,
The files look correct and that’s what I’ve been using but I haven’t had any issues at all anymore and I’ve run some load tests with upwards of 3000 users, 2000 rooms and a few million messages on a single node and while Openfire was pretty busy of course, it never failed over again. I was able to continue to login new users manually via my XMPP client.
Also when you say Openfire was at 100% CPU, were you still able to login new users or was everything locked up?
If you’re able to reproduce the issue, can you run jstack to get the thread dump? (/bin/jstack -F -l > openfire_jstack.log)
My Openfire server seems to be responding normally in that users are able to log in/ send messages/ join/create rooms. It’s just that there’s still a spinning process.
It’s quite possible that it’s a different issue
I hadn’t restarted my server yet so I tried running jstack- the only output to file I got (running it as root) was:
Deadlock Detection:
Can’t print deadlocks:Unable to deduce type of thread from address 0x00007f51e4006800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
Trying it as the daemon user caused the server to hang in a “T” state and ultimately required a restart of the openfire service to resume. I’ll try another jstack if/when it happens again.
Does anyone else running it on linux has same openfire.jar.mina file? I don’t, but i’m on Windows.
That file is non-standard of my creation, I tried one of the buils from tarball previously and moved it aside by renaming it when I learned it didn’t solve the issue.
I’m happy to delete it.
Finally one quick question for you- what JRE/JDK are you running? For instance should I try an Oracle JDK instead of OpenJDK, or move to Java 1.8 from 1.7?
In fact I’ve just gone ahead and deleted it anyway and restarted
I run my test server with a built in Oracle JRE (in windows installer), which is 1.7.0_76, but i occasionally will delete it and run with a system’s Java 1.8.0_latest (Oracle again). My production server is always on Java 8 latest. But it is still running 3.9.3.
Have installed the latest RC, and while I’m still seeing 100% CPU on the java process, it’s only using up one core, and people are still connected to the server 12hrs later.
edit: spoke too soon. whilst it’s more stable, generally speaking, than 3.10.0 - around an hour ago it started to kick people out and refuse to accept new connections.
So far so good, it’s survived the weekend and one day in use without the CPU usage spinning up.
Version 3.10 don´t work, I used 3.8.1 and works fine.
“-Djava.net.preferIPv4Stack=true” = it worked less than hour, CPU usage decreased, but after ± 40 clients (sparks) connected, others new connections are refused.
Web Console don´t log, too. And I need to “stop” and “start” openfire, because restart don´t work. I’m trying to find the version 3.9.3 to download and downgrade, we are several days without spark.
We have released 3.10.1 now, please do try it. I am inclined to close this thread as it has gotten too long to be useful, so please consider opening new threads for issues found with 3.10.1 release.
1 Like
Yeah, you should probably lock it.
All, Openfire 3.10.2 is now released, which we are more optimistic than ever as to having resolved this “100% CPU” issue. Please open new discussion threads if you continue to see this!
Hi,
1 Like
Any actually solution for this 100% cpu usage? its incredible that this post has about 175 reply… please…
my configuration its on windows server 2012 r2
hello I’m facing the same problem with clean install openfire latest version 4.4.0 64 bits bundled JRE
ms sql server database
windows server 2012 r2
64 gb ram
AD intregation
The program start ok but in 1 hour the preocess start peak at 100%
Any idea why?
I have search the internet for this and there are lots of peoples facing the same problem
where is the solution??