JVM Settings and Debugging

  • Useful Links
    • Virtual Machines
    • Tools
  • Settings for Openfire
    • Disable IPv6
    • Heap Settings
    • Monitor the JVM Memory Usage
    • Heap dump on OutOfMemory error
    • DNS Settings
    • UTF-8 Settings
  • Debugging
    • Creating a stack trace
    • Thread identification
      • Linux native to java thread mapping
      • Windows native to java thread mapping
  • More advanced debugging and profiling
    • Tools
      • Remote debugging of Openfire
      • Remote profiling of Openfire with Profiler4J
      • Random disconnects on high load or during load tests

Useful Links

Virtual Machines

Tools

Settings for Openfire

Openfire Installation Guide (Visit this page if you wonder how to set these values.)

Disable IPv6

As still some users do not use IPv6, it is save to disable it by adding as the third line in bin/openfire:

INSTALL4J_ADD_VM_PARAMS="-Djava.net.preferIPv4Stack=true"

Heap Settings

Add “-Xms32m -Xmx128m -Xss128k -Xoss128k -XX:ThreadStackSize=128” to INSTALL4J_ADD_VM_PARAMS to specify an initial heap of 32 MB, a maximum heap of 128 MB and a stack size of 128 kB for native and java threads. Do not append a k to the ThreadStackSize value.

One may also need to set “-XX:MaxPermSize=128m”, but before doing this one should consult the gc.log file.

Monitor the JVM Memory Usage

Add “-XX:+PrintGCDetails -Xloggc:/tmp/gc.log” to INSTALL4J_ADD_VM_PARAMS to track every garbage collecion and allocation failure.

Heap dump on OutOfMemory error

Add “-XX:+HeapDumpOnOutOfMemoryError” to INSTALL4J_ADD_VM_PARAMS to create a heap dump every time an OutOfMemory error occurs. There should never occur one, but if it does the dump contains useful information to debug the issue.

Note: If you did not adjust the heap settings and try to connect 10000 users to Openfire then you will also get a heap dump which is not really interesting.

DNS Settings

If s2s connections to servers with dynamic IP addresses are possible one may want to change the DNS cache to a low value instead of forever. One can set the “networkaddress.cache.ttl=1800” value in

“path-to-JRE\lib\security\java.security” (Windows) / “path-to-JRE/lib/security/java.security” (Linux) or add “-Dsun.net.inetaddr.ttl=1800” as an additional INSTALL4J_ADD_VM_PARAMS parameter.

“1800” seconds are 30 minutes, this should be a fine value.

JM-898 should fix this issue.

UTF-8 Settings

Add “-Dfile.encoding=UTF-8” to the INSTALL4J_ADD_VM_PARAMS parameter. This fixes a few issues: JM-1351, JM-1190, JM-1101.

Debugging

To identify deadlocks or looping threads you need to create a stack trace.

Creating a stack trace

Use `jstack openfire-pid >/tmp/javacore.txt´ to write one into /tmp/javacore.txt.

On Linux `kill -3 openfire-pid´ will cause the JVM to write one to STDOUT, one will likely find in in nohup.out, stdout.log or somewhere else, depending on your start script.

One can also use StackTrace (graphical UI) to create one, the link is at the top of the page.

On Windows all methods may fail if the process runs as SYSTEM and your user does not have SYSTEM rights.

Thread identification

To identify looping threads you should also get one or more stack traces and try to identify the looping thread.

Linux native to java thread mapping

This short manual is for Linux and it may not work on your Linux installation, depending on your kernel and ps versions:

You may want to try to run `ps -T -p openfire-pid -o pid,tid,pri,time | grep -v '00:00:00’´ - this could (depending on your linux ps implementation) display the thread which is using a lot of CPU just like:

PID TID PRI TIME

8313 8313 23 00:00:01

8313 8314 21 00:13:23 <== high “time” value, look at the TID, I use 8314 as a sample value.

8313 8315 23 00:00:04

… very long list, run the command above or `ps -T -p openfire-pid -o pid,tid,pri,time | grep 8314´ again to make sure that this value increases.

Then you would have to look for the “tid” with the highest “time” value and convert it from dec to hex (8314=0x207A ; Linux: printf "%X\n" 8314´; Windows:calc´ with scientific view).

You should find somewhere in the javacore a line with “… nid=0x207A …” - this should be the looping thread.

Windows native to java thread mapping

Get “pslist” from MS Technet, Process Utilities, PsList.

Run “pslist.exe -dmx Openfire >c:\temp\openfire.threads.txt” to create a list of threads within Openfire. Also the Openfire PID will be written to the file (you can alternatively look in the Task Manager for it).

Run “jstack openfire-pid >c:\temp\openfire.javacore.txt” to create a core file. This may fail depending on service permissions - for Spark it usually works fine.

Then you need to map the interesting “Tid” in openfire.threads.txt´ to a "nid" inopenfire.javacore.txt´ by converting it to hex(Tid 252 ==> nid=0xFC).

More advanced debugging and profiling

Probably only useful for developers. Do not use these options unless you know what you are doing.

Tools

Remote debugging of Openfire

Start Openfire with these JVM options: “-Xdebug -agentlib:jdwp=transport=dt_socket,server=y,address=8000”

Details:http://java.sun.com/j2se/1.5.0/docs/guide/jpda/conninv.html

Remote profiling of Openfire with Profiler4J

Download the CVS source and build profiler4j-1.0-beta3. Add “-javaagent:/path-to/agent.jar=waitconn=false” to the JVM options. Then you need to start also the Profiler4J console to get profiling information.

Random disconnects on high load or during load tests

Use the loadstats plugin, either compile it with Openfire or download it here Load Stats Plugin

The wiki document which describes its output is as far as I can tell no longer available.

Where do you add the heap values for a linux install?

Second that question - where in a Linux install is the INSTALL4J_ADD_VM_PARAMS value actually set? On my system I have added it to the file /etc/sysconfig/openfire - but I don’t know if this is correct and therefore if it’s having any effect.

Thanks, Nick

OK what I’ve done is add the extra heap values to OPENFIRE_OPTS (I’ve ignored INSTALL4J_ADD_VM_PARAMS) and now when I run ps -ef | grep openfire I can see that the values have indeed been added. So I reckon this is where they should go.

Nick

Well, and how do you add it in windows?

Thanks in advance, Maddi

Edit: found it

For thoose who are as blind as me: http://www.igniterealtime.org/builds/openfire/docs/latest/documentation/install- guide.html#windows

Hello

I’ve installed Openfire with the .deb on Ubuntu 8.04 and i can’t find /etc/sysconfig/openfire.

Could you please help me?

Many thx

If you install Openfire using Debian package, it should be in /etc/init.d/openfire. if you install Beta version from .tar.gz you should create this file by adding a symbolic link to openfire.sh