Openfire server stopped responding

Openfire Openfire Support
1

We are running open fire in a windows server. Suddenly it stopped responding and the error log has the following error.

org.jivesoftware.openfire.nio.ConnectionHandler - Closing connection due to error while processing message: AAA=

java.util.NoSuchElementException

at java.util.StringTokenizer.nextToken(Unknown Source)

at org.jivesoftware.openfire.sasl.SaslServerPlainImpl.evaluateResponse(SaslServerP lainImpl.java:114)

at org.jivesoftware.openfire.net.SASLAuthentication.handle(SASLAuthentication.java :274)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:179)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:181)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

There was one more file in the bin folder of open fire with name ‘hs_err_pid1888.log’ and the following content was found

There is insufficient memory for the Java Runtime Environment to continue.

Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate

Possible reasons:

The system is out of physical RAM or swap space

In 32 bit mode, the process size limit was hit

Possible solutions:

Reduce memory load on the system

Increase physical memory or swap space

Check if swap backing store is full

Use 64 bit Java on a 64 bit OS

Decrease Java heap size (-Xmx/-Xms)

Decrease number of Java threads

Decrease Java thread stack sizes (-Xss)

Set larger code cache with -XX:ReservedCodeCacheSize=

This output file may be truncated or incomplete.

Out of Memory Error (allocation.cpp:211), pid=1888, tid=3068

JRE version: Java™ SE Runtime Environment (7.0_51-b13) (build 1.7.0_51-b13)

Java VM: Java HotSpot™ Client VM (24.51-b03 mixed mode windows-x86 )

Failed to write core dump.

--------------- T H R E A D ---------------

Current thread (0x486bfc00): VMThread [stack: 0x01ea0000,0x01ef0000] [id=3068]

Stack: [0x01ea0000,0x01ef0000], sp=0x01eeeb74, free space=314k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V [jvm.dll+0x18e0f1]

V [jvm.dll+0x1a39a5]

V [jvm.dll+0x1a4b8b]

C [MSVCR100.dll+0x5c556]

C [MSVCR100.dll+0x5c600]

C [kernel32.dll+0x1338a]

C [ntdll.dll+0x39f72]

C [ntdll.dll+0x39f45]

VM_Operation (0x5a14ef28): BulkRevokeBias, mode: safepoint, requested by thread 0x71863400

When this crash happened the server memory was at 30% and cpu usage was negligible and we have set max heap size as ‘-Xmx1024m’ in openfire-service.vmoptions file and we are not specifying min heap size, not sure what has caused this, any help?

Thanks,

Pavan

2

This problem is not specific to Openfire, it’s a very generic “out of memory” problem. Most likely, your operating system did not have enough memory available for the Openfire heap. Try reducing the maximum heapsize (using -Xmx, but with a lower value) or close other applications on the same server. The error message that you quoted contains a number of other possible solutions.

3

Thanks for your reply, Guus. The jvm heap size is set to 1GB while the total available memory on the server is 8GB. Also, at the time when this issue happened, system memory was not under pressure at all; server monitoring logs indicate that more than 50% memory was free. What surprises me is how come OS fails to do native memory allocation when it has so much free memory at it’s disposal. What you suggest would make sense if there were indications of OS running out of memory but that doesn’t seem to be the case when 4 GB memory (out of 7GB non-heap memory) was still free and available for OS. Could there be anything else going wrong here?

4

Well, I can quote part of the error message for you:

There is insufficient memory for the Java Runtime Environment to continue. Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate

Possible reasons:

  • The system is out of physical RAM or swap space

  • In 32 bit mode, the process size limit was hit
    Possible solutions:

  • Reduce memory load on the system

  • Increase physical memory or swap space

  • Check if swap backing store is full

  • Use 64 bit Java on a 64 bit OS

  • Decrease Java heap size (-Xmx/-Xms)

  • Decrease number of Java threads

  • Decrease Java thread stack sizes (-Xss)

  • Set larger code cache with -XX:ReservedCodeCacheSize=

5

Thanks Guus… I’m reading up on memory management on java side and based on that knowledge I’ll review your reply and comment.