Troubleshoot server crashing

Openfire Openfire Support
1

We had a version of 3.1.1 running without any issues until recently. We attempted to upgrade to 3.2.2 but had some problems (not recall exactly what) so we brought back the 3.1.1 version and can’'t seem to be able to keep the server running. The server usually crashes within seconds, I am attaching 3 partial log files in the event that it may help and some server information.

Linux Distribution: Ubuntu 6.06 LTS

java version “1.5.0_06”

Java™ 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)

Java HotSpot™ Client VM (build 1.5.0_06-b05, mixed mode, sharing)

Added the following line to my wildfire script, following the wiki documentation:

INSTALL4J_ADD_VM_PARAMS="-Djava.net.preferIPv4Stack=true -Xms32m -Xmx128m -Xss128k -Xoss128k -XX:ThreadStackSize=128 -XX:+PrintGCDetails -Xloggc:/opt/wildfire/logs/gc.log"

  1. An unexpected error has been detected by HotSpot Virtual Machine:

  1. SIGSEGV (0xb) at pc=0xb7592eb7, pid=5036, tid=2786577328

  1. Java VM: Java HotSpot™ Server VM (1.5.0_06-b05 mixed mode)

  2. Problematic frame:

  3. V

T H R E A D -

Current thread (0x08108b38): JavaThread “CompilerThread0” daemon

siginfo:si_signo=11, si_errno=0, si_code=2, si_addr=0x083d8e94

Registers:

EAX=0x083d8e90, EBX=0xb79d37f0, ECX=0xb79cabc8, EDX=0xa617b390

ESP=0xa617adbc, EBP=0xa617add8, ESI=0x083e15f4, EDI=0x08140db8

EIP=0xb7592eb7, CR2=0x083d8e94, EFLAGS=0x00010292

Top of Stack: (sp=0xa617adbc)

0xa617adbc: b78ffa27 083d8e90 00000080 a617adf8

0xa617adcc: b78ffa0d b79d37f0 a617ae40 a617adf8

0xa617addc: b750b362 083e15f4 00000080 00000080

0xa617adec: b79d37f0 a617aeb0 a617ae40 a617ae28

0xa617adfc: b74f5d75 08140e68 00000400 00000080

0xa617ae0c: 00000080 00000080 000000a0 00000001

0xa617ae1c: b79d37f0 08140db8 a617ae40 a617aee8

0xa617ae2c: b780a17b 08140db8 a617ae40 a617b390

Instructions: (pc=0xb7592eb7)

0xb7592ea7: c1 f8 02 c3 90 8d 74 26 00 55 89 e5 8b 45 08 5d

0xb7592eb7: 8b 40 04 c3 90 8d 74 26 00 55 31 c0 89 e5 8b 55

Stack: [0xa60fc000,0xa617d000), sp=0xa617adbc, free space=507k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V

V

V

V

V

V

V

V

V

V

V

V

V

V

V

C

Current CompileTask:

opto: 43 com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read([CII)I (1385 bytes)

  1. An unexpected error has been detected by HotSpot Virtual Machine:

  1. SIGSEGV (0xb) at pc=0x73656363, pid=4899, tid=2786638768

  1. Java VM: Java HotSpot™ Server VM (1.5.0_06-b05 mixed mode)

  2. Problematic frame:

  3. C 0x73656363

T H R E A D -

Current thread (0x08109ba8): JavaThread “CompilerThread1” daemon

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x73656363

Registers:

EAX=0x08336ec4, EBX=0xb7a687f0, ECX=0xb72be7a8, EDX=0x0845f208

ESP=0xa6189cec, EBP=0xa6189d38, ESI=0x08338158, EDI=0x08338108

EIP=0x73656363, CR2=0x73656363, EFLAGS=0x00010292

Top of Stack: (sp=0xa6189cec)

0xa6189cec: b78af295 08336ec4 a618a240 00000001

0xa6189cfc: b790bc3c 083380dc 080a8038 a6189d38

0xa6189d0c: b790bdd5 08338108 080a7d88 a618a25c

0xa6189d1c: a618a25c 000000bc 08338138 08338254

0xa6189d2c: b7a687f0 a618a240 08338108 a6189d88

0xa6189d3c: b790e1fd 08338108 a618a240 00000001

0xa6189d4c: 083380dc 0845ebd0 a6189ef0 08338108

0xa6189d5c: 0845ee71 0845f0f8 00000c39 080a8038

Instructions: (pc=0x73656363)

0x73656353:

Stack: [0xa610b000,0xa618c000), sp=0xa6189cec, free space=507k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

C 0x73656363

V

V

V

V

V

V

V

V

V

V

V

C

Current CompileTask:

opto: 26 sun.security.provider.SHA.implCompress([BI)V (494 bytes)

  1. An unexpected error has been detected by HotSpot Virtual Machine:

  1. SIGSEGV (0xb) at pc=0xb74a75fd, pid=4730, tid=2852027312

  1. Java VM: Java HotSpot™ Server VM (1.5.0_06-b05 mixed mode)

  2. Problematic frame:

  3. V

T H R E A D -

Current thread (0x08109858): JavaThread “CompilerThread1” daemon

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0aaf9df0

Registers:

EAX=0x0088036e, EBX=0xb79a47f0, ECX=0x0aaf9df0, EDX=0x088f9038

ESP=0xa9fe5d04, EBP=0xa9fe5d48, ESI=0x00000002, EDI=0xffffffff

EIP=0xb74a75fd, CR2=0x0aaf9df0, EFLAGS=0x00010296

Top of Stack: (sp=0xa9fe5d04)

0xa9fe5d04: 00000000 a9fe5d30 00000000 088ce428

0xa9fe5d14: 088ce428 b2494108 00000000 00000000

0xa9fe5d24: 00000000 00000000 00000001 00000000

0xa9fe5d34: 00000002 ffffffff b79a47f0 a9fe67c0

0xa9fe5d44: a9fe6210 a9fe5d68 b74a81d2 08885ea8

0xa9fe5d54: 00000000 a9fe6210 00000001 b7949508

0xa9fe5d64: b79a47f0 a9fe5df8 b7822005 08885ea8

0xa9fe5d74: a9fe6210 00000000 00000000 086ee65c

Instructions: (pc=0xb74a75fd)

0xb74a75ed: 89 7d f0 0f bf 70 02 89 75 ec 8b 41 1c 8d 0c 82

0xb74a75fd: 0f bf 31 89 75 e8 0f bf 51 02 57 89 55 e4 e8 f0

Stack: [0xa9f67000,0xa9fe8000), sp=0xa9fe5d04, free space=507k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V

V

V

V

V

V

V

V

V

V

V

V

V

C

Current CompileTask:

opto:121 ! java.lang.Throwable.printStackTrace(Ljava/io/PrintWriter;)V (87 bytes)

2

Hi,

did adding the parameters cause trouble for you or the fact that you were running 3.2.2 and did downgrade to 3.1.1?

It may help a lot to use only

INSTALL4J_ADD_VM_PARAMS="-Djava.net.preferIPv4Stack=true -Xms32m -Xmx128m"

or if you really need the GC log

INSTALL4J_ADD_VM_PARAMS="-Djava.net.preferIPv4Stack=true -Xms32m -Xmx128m -XX:+PrintGCDetails -Xloggc:/opt/wildfire/logs/gc.log"

It may cause trouble to modify the native initial and max stack size using the other parameters.

If you still get compile errors you could disable the Java compiler with the “-Xint” option - this means of course that your server will be slow, but it should no longer crash with a “Current CompileTask” error.

LG

3

Hi LG,

Thanks for your input. I modified the INSTALL4J_ADD_VM_PARAMS line per your recommendation, without the GC logging, without an improvement. The server would be up for a slightly longer period of time but would consistently crash time and time again.

I am starting to lean toward an issue with either the hardware or the JVM itself but I am not sure how to start troubleshooting that part of the equation. While attempting to read the latest log file I received the following message when calling vim

Vim: Caught deadly signal SEGV] 330L, 21204C

Vim: Finished.

Vim: Double signal, exiting

Segmentation fault

Minutes later my ssh connection dropped and the server’'s monitor showed

swap_free: Bad swap offset entry 00039000

swap_free: unused swap offset entry 00003800

Maybe it is time to look beyond wildfire as the root cause.