- Useful Links
- Virtual Machines
- Tools
- Settings for Openfire
- Disable IPv6
- Heap Settings
- Monitor the JVM Memory Usage
- Heap dump on OutOfMemory error
- DNS Settings
- UTF-8 Settings
- Debugging
- Creating a stack trace
- Thread identification
- Linux native to java thread mapping
- Windows native to java thread mapping
- More advanced debugging and profiling
- Tools
- Remote debugging of Openfire
- Remote profiling of Openfire with Profiler4J
- Random disconnects on high load or during load tests
- Tools
Useful Links
Virtual Machines
- Sun - http://java.sun.com/javase/downloads/index.jsp
- IBM - http://www-128.ibm.com/developerworks/java/jdk/
- Bea JRockit - http://commerce.bea.com/products/weblogicjrockit/jrockit_prod_fam.jsp
Tools
- GCViewer - http://www.tagtraum.com/gcviewer.html
- http://www.tagtraum.com/ contains also a lot of the parameters one may add to a JVM.
- StackTrace - http://www.adaptj.com/root/main/stacktrace
- Sun JDK: JStack (jstack.exe on Windows)
- Sun JDK: JConsole - http://java.sun.com/javase/6/docs/technotes/guides/management/jconsole.html (in the bin directory of the JDK)
- VisualJVM - https://visualvm.dev.java.net/ - create stack traces, heap dumps, …
- Profiler4J
Settings for Openfire
Openfire Installation Guide (Visit this page if you wonder how to set these values.)
Disable IPv6
As still some users do not use IPv6, it is save to disable it by adding as the third line in bin/openfire:
INSTALL4J_ADD_VM_PARAMS="-Djava.net.preferIPv4Stack=true"
Heap Settings
Add “-Xms32m -Xmx128m -Xss128k -Xoss128k -XX:ThreadStackSize=128” to INSTALL4J_ADD_VM_PARAMS to specify an initial heap of 32 MB, a maximum heap of 128 MB and a stack size of 128 kB for native and java threads. Do not append a k to the ThreadStackSize value.
One may also need to set “-XX:MaxPermSize=128m”, but before doing this one should consult the gc.log file.
Monitor the JVM Memory Usage
Add “-XX:+PrintGCDetails -Xloggc:/tmp/gc.log” to INSTALL4J_ADD_VM_PARAMS to track every garbage collecion and allocation failure.
Heap dump on OutOfMemory error
Add “-XX:+HeapDumpOnOutOfMemoryError” to INSTALL4J_ADD_VM_PARAMS to create a heap dump every time an OutOfMemory error occurs. There should never occur one, but if it does the dump contains useful information to debug the issue.
Note: If you did not adjust the heap settings and try to connect 10000 users to Openfire then you will also get a heap dump which is not really interesting.
DNS Settings
If s2s connections to servers with dynamic IP addresses are possible one may want to change the DNS cache to a low value instead of forever. One can set the “networkaddress.cache.ttl=1800” value in
“path-to-JRE\lib\security\java.security” (Windows) / “path-to-JRE/lib/security/java.security” (Linux) or add “-Dsun.net.inetaddr.ttl=1800” as an additional INSTALL4J_ADD_VM_PARAMS parameter.
“1800” seconds are 30 minutes, this should be a fine value.
JM-898 should fix this issue.
UTF-8 Settings
Add “-Dfile.encoding=UTF-8” to the INSTALL4J_ADD_VM_PARAMS parameter. This fixes a few issues: JM-1351, JM-1190, JM-1101.
Debugging
To identify deadlocks or looping threads you need to create a stack trace.
Creating a stack trace
Use `jstack openfire-pid >/tmp/javacore.txt´ to write one into /tmp/javacore.txt.
On Linux `kill -3 openfire-pid´ will cause the JVM to write one to STDOUT, one will likely find in in nohup.out, stdout.log or somewhere else, depending on your start script.
One can also use StackTrace (graphical UI) to create one, the link is at the top of the page.
On Windows all methods may fail if the process runs as SYSTEM and your user does not have SYSTEM rights.
Thread identification
To identify looping threads you should also get one or more stack traces and try to identify the looping thread.
Linux native to java thread mapping
This short manual is for Linux and it may not work on your Linux installation, depending on your kernel and ps versions:
You may want to try to run `ps -T -p openfire-pid -o pid,tid,pri,time | grep -v '00:00:00’´ - this could (depending on your linux ps implementation) display the thread which is using a lot of CPU just like:
PID TID PRI TIME
8313 8313 23 00:00:01
8313 8314 21 00:13:23 <== high “time” value, look at the TID, I use 8314 as a sample value.
8313 8315 23 00:00:04
… very long list, run the command above or `ps -T -p openfire-pid -o pid,tid,pri,time | grep 8314´ again to make sure that this value increases.
Then you would have to look for the “tid” with the highest “time” value and convert it from dec to hex (8314=0x207A ; Linux: printf "%X\n" 8314´; Windows:
calc´ with scientific view).
You should find somewhere in the javacore a line with “… nid=0x207A …” - this should be the looping thread.
Windows native to java thread mapping
Get “pslist” from MS Technet, Process Utilities, PsList.
Run “pslist.exe -dmx Openfire >c:\temp\openfire.threads.txt” to create a list of threads within Openfire. Also the Openfire PID will be written to the file (you can alternatively look in the Task Manager for it).
Run “jstack openfire-pid >c:\temp\openfire.javacore.txt” to create a core file. This may fail depending on service permissions - for Spark it usually works fine.
Then you need to map the interesting “Tid” in openfire.threads.txt´ to a "nid" in
openfire.javacore.txt´ by converting it to hex(Tid 252 ==> nid=0xFC).
More advanced debugging and profiling
Probably only useful for developers. Do not use these options unless you know what you are doing.
Tools
- JConsole - http://java.sun.com/javase/6/docs/technotes/guides/management/jconsole.html (in the bin directory of the JDK)
- VisualJVM - https://visualvm.dev.java.net/ - create stack traces, heap dumps, …
- Profiler4J
Remote debugging of Openfire
Start Openfire with these JVM options: “-Xdebug -agentlib:jdwp=transport=dt_socket,server=y,address=8000”
Details:http://java.sun.com/j2se/1.5.0/docs/guide/jpda/conninv.html
Remote profiling of Openfire with Profiler4J
Download the CVS source and build profiler4j-1.0-beta3. Add “-javaagent:/path-to/agent.jar=waitconn=false” to the JVM options. Then you need to start also the Profiler4J console to get profiling information.
Random disconnects on high load or during load tests
Use the loadstats plugin, either compile it with Openfire or download it here Load Stats Plugin
The wiki document which describes its output is as far as I can tell no longer available.