Openfire service crashing daily

Openfire Openfire Support
1

The service will run for a day, maybe two, then it will stop with the error message below. The error is identical to post http://www.igniterealtime.org/community/message/198731#198731. I posted a response in that forum 6 days ago with no response. I figured I try creating my own thread. I also posted a thread in http://kraken.blathersource.org/node/266. I have a feeling the problems may be related.

We are running Openfire 3.6.4 w/ 1.6.0_17 JVM

Windows Server 2003 SP2 (Virtual Server in VMware ESX)

Dual 2.3Ghz Procs / 3GB of RAM

-Xms1280m -Xmx1280m (You can see all arguments in the error msg below. We tried 256/256, 512/512, 1024/1024, 1024/1536 and 512/1024 as recommended in the linked article…same problems)

Plugins:

Client Control: Only allow Spark client

Kraken IM Gateway 1.1.2: We control IM Gateway access through groups. Only 170 users are allowed to used it.

Monitoring Service: Full archiving is enabled. User to User and group chats

Red5: Used for video conferencing test (3 users) and where JWCHAT (~200 JWCHAT users) is setup . HTTP Binding is enabled for this.

Search: Used for easy searching…duh

User Import Export: Used for migration of user from old IM solution

We have 3000 accounts w/ a max of 1500 concurrent users. We average 1300 to 1400 concurrent users.

Running SQL 2005 DB (on a dedicated SQL server)

Microsoft SQL Server Management Studio 9.00.4035.00
Microsoft Analysis Services Client Tools 2005.090.4035.00
Microsoft Data Access Components (MDAC) 2000.086.3959.00 (srv03_sp2_rtm.070216-1710)
Microsoft MSXML 2.6 3.0 6.0
Microsoft Internet Explorer 7.0.5730.13
Microsoft .NET Framework 2.0.50727.3082
Operating System 5.2.3790

Error Message…I excluded the process list due to length. If this will help, I can post it. If anyone has a solution, I am in desperate need of help. Everything worked great during our testing of 400-500 concurrent users.

An unexpected error has been detected by Java Runtime Environment:

java.lang.OutOfMemoryError: requested 835968 bytes for Chunk::new. Out of swap space?

Internal Error (414C4C4F434154494F4E0E43505000C7), pid=3188, tid=3168

Java VM: Java HotSpot™ Server VM (1.6.0_03-b05 mixed mode)

If you would like to submit a bug report, please visit:

http://java.sun.com/webapps/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread (0x4801f800): JavaThread “CompilerThread1” daemon [_thread_in_native,]

Stack: [0x483b0000,0x48400000)
[error occurred during error reporting, step 110, id 0xc0000005]

Current CompileTask:
C2:2692 org.jivesoftware.openfire.plugin.SearchPlugin.replyDataFormResult(Ljava/util/Co llection;Lorg/xmpp/packet/IQ;)Lorg/xmpp/packet/IQ; (416 bytes)

VM Arguments:
jvm_args: -Dexe4j.isInstall4j=true -Dexe4j.isService=true -Dexe4j.moduleName=C:\Program Files\Openfire\bin\openfire-service.exe -Dexe4j.processCommFile=C:\WINDOWS\TEMP\e4j_p3188.tmp -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Dexe4j.consoleCodepage=cp0 -Xrs -Xms1024m -Xmx1024m
java_command:
Launcher Type: generic

Environment Variables:
PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;c:\program files\openfire\jre\bin
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel

--------------- S Y S T E M ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 2096584k(1255816k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot™ Server VM (1.6.0_03-b05) for windows-x86, built on Sep 24 2007 22:20:35 by “java_re” with unknown MS VC++:1310

We get the errors I posted below for a few hours then the OpenFire admin console will stop responding for 30-45 minutes, then the service crashes.

------------------------------------------------------------------error--------- ---------------------------------------------------

2009.12.16 11:36:00 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)
]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at net.kano.joscar.flap.AsynchronousFlapProcessor.(AsynchronousFlapProcessor .java:29)
at net.kano.joscar.flap.ClientFlapConn.init(ClientFlapConn.java:82)
at net.kano.joscar.flap.ClientFlapConn.(ClientFlapConn.java:73)
at net.sf.kraken.protocols.oscar.AbstractFlapConnection.(AbstractFlapConnect ion.java:102)
at net.sf.kraken.protocols.oscar.LoginConnection.(LoginConnection.java:41)
at net.sf.kraken.protocols.oscar.OSCARSession.logIn(OSCARSession.java:120)
at net.sf.kraken.protocols.oscar.OSCARTransport.registrationLoggedIn(OSCARTranspor t.java:95)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:398)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:199)
at org.jivesoftware.openfire.component.InternalComponentManager$RoutableComponents .process(InternalComponentManager.java:619)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:260)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:164)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java: 337)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:176)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)

2009.12.16 11:35:51 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: MSG 59 U 92

 Chunk Debug

====================
MIME-Version: 1.0
Content-Type: text/x-msmsgscontrol
TypingUser: @hotmail.com

====================
Binary Chunk Debug

00000000h: 4D 49 4D 45 2D 56 65 72 73 69 6F 6E 3A 20 31 2E ; MIME-Version: 1.
00000010h: 30 0D 0A 43 6F 6E 74 65 6E 74 2D 54 79 70 65 3A ; 0…Content-Type:
00000020h: 20 74 65 78 74 2F 78 2D 6D 73 6D 73 67 73 63 6F ; text/x-msmsgsco
00000030h: 6E 74 72 6F 6C 0D 0A 54 79 70 69 6E 67 55 73 65 ; ntrol…TypingUse
00000040h: 72 3A 20 6A 72 61 31 39 37 36 40 68 6F 74 6D 61 ; r: @hotma
00000050h: 69 6C 2E 63 6F 6D 0D 0A 0D 0A 0D 0A ; il.com…

java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)

------------------------------------------------------------------error--------- ---------------------------------------------------

2

Hi,

your “unable to create new native thread” error shows that the Java heap (Xmx) is not your problem. You may want to try to set “-XX:ThreadStackSize=128” to have more space for native threads.

LG

3

Would this be set in the **openfire-service.vmoptions **file also? I found this article (http://www.igniterealtime.org/community/docs/DOC-1033). What would be recommended settings for “-Xss128k -Xoss128k -XX:ThreadStackSize=128” for 2000 users. I understand it depends on plugins and what the users are doing… I already have -Xms1280 and -Xmx1280 set. What would be required to set - Xss or -Xoss based on our load/configuration? If this is to subjective to suggest settings, how would I determine what’s appropriate

I have this posted in the Kraken forums also. Just in case it’s a problem with the plugin.

http://kraken.blathersource.org/node/266

We where running 1.1.2 and updated it to 1.1.3. The service ran for the entire following day and the weekend (less load). Come Monday (yesterday) morning the Openfire console wasn’t working but the port was responding. IM was still working… The server CPU was pegged to 100%. I was off so a co-worker just rebooted the server. I would assumed it would have crashed again if he wouldn’t have… ?

Today so far, so good. 1500 connections under 18% Java memory (-Xms1280m, -Xmx1280m).

If you need me to post my logs or JVM graphs, please let me know. Our users are not to happy with the constant service crashes.

We’ve been getting tons of these messages…

2009.12.21 10:39:58 [org.jivesoftware.openfire.event.SessionEventDispatcher.dispatchEvent(SessionEv entDispatcher.java:87)
]
java.lang.IllegalStateException: Timer already cancelled.
at java.util.Timer.sched(Unknown Source)
at java.util.Timer.schedule(Unknown Source)
at org.jivesoftware.openfire.plugin.spark.TaskEngine.schedule(TaskEngine.java:98)
at org.jivesoftware.openfire.plugin.spark.SparkManager$SparkSessionListener.sessio nCreated(SparkManager.java:373)
at org.jivesoftware.openfire.event.SessionEventDispatcher.dispatchEvent(SessionEve ntDispatcher.java:63)
at org.jivesoftware.openfire.SessionManager.addSession(SessionManager.java:528)
at org.jivesoftware.openfire.session.LocalClientSession.setAuthToken(LocalClientSe ssion.java:589)
at org.jivesoftware.openfire.handler.IQBindHandler.handleIQ(IQBindHandler.java:141 )
at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:49)
at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:351)
at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:101)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)
at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:319)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:284)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:176)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)

4

We resolved this issue by telling people to not use Empathy/Telepathy based IM clients for now (which are apparently the default with the latest release of Ubuntu Linux). Since doing this we’ve been running w/o problem for almost a week.

I’m monitoring threads the Empathy/Telepathy and Openfire forums to watch for a fix, but none yet.

This search shows lots of threads in the forums related to this topic:

http://www.igniterealtime.org/community/search.jspa?peopleEnabled=true&userID=&c ontainerType=&container=&q=empathy

5

Thanks for the reply… I have seen those postings and we run only the Spark and JWChat web client.

6

I would suggest disabling some of the plugins you are running. They may be the cause of your issues. Then enable them one at a time. Or remove them one at a time. Start with removing client control.

7

Hi,

the thread stack size applies to every thread and 128k may be too small - you’ll get Java errors in your log file if this is the case. Xss should be the same as ThreadStackSize and thus for the native stack size while Xoss should set the java stack size for every thread. As you have a problem with the native thread count setting the ThreadStackSize should be enough.

You may want to create some stack traces while the server is running - there you may be able to see an increasing number of java threads (every java thread can be mapped to a native thread). Maybe the name helps you to identify which part of Openfire or plugins does start these threads.

LG

8

I added “-XX:ThreadStackSize=128” and set xmpp.pep.enabled=false (just incase). The service appears to have ran longer (may be due to the holiday) but we still are having problems. Today the service didn’t crash but you couldn’t get into the web client or the admin console. The server CPU was pegged at 100% and the following event was logged in the eventvwr.

Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2019
Date: 12/30/2009
Time: 9:08:59 AM
User: N/A
Computer:
Description:
The server was unable to allocate from the system nonpaged pool because the pool was empty.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 04 00 01 00 54 00 …T.
0008: 00 00 00 00 e3 07 00 c0 …ã…À
0010: 00 00 00 00 9a 00 00 c0 …�…À
0018: 00 00 00 00 00 00 00 00 …
0020: 00 00 00 00 00 00 00 00 …
0028: 08 00 00 00 …

I also noticed out Statistics page is not showing up…it’s there, but empty (http://www.igniterealtime.org/community/thread/36238 and http://www.igniterealtime.org/community/message/172876#172876). We are going to attempt to repair the ofRRDs table this weekend. Could this be the source of all my crashes?

Here is my error.log around that time…

2009.12.30 09:01:32 [org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceMana gerImpl.java:271)
] Error storing offline presence of user:
java.sql.SQLException: Violation of PRIMARY KEY constraint ‘ofPresence_pk’. Cannot insert duplicate key in object ‘dbo.ofPresence’.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2816)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2254)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:631)
at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.execute(JtdsPreparedStatement.j ava:559)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.logicalcobwebs.proxool.ProxyStatement.invoke(ProxyStatement.java:100)
at org.logicalcobwebs.proxool.ProxyStatement.intercept(ProxyStatement.java:57)
at $java.sql.Wrapper$$EnhancerByProxool$$b2f44fdd.execute()
at org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceManag erImpl.java:268)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:145)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:112)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:176)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:134)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1133)
at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:202)
at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:185)
at org.jivesoftware.openfire.nio.ConnectionHandler.sessionClosed(ConnectionHandler .java:87)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.sessionClosed(A bstractIoFilterChain.java:550)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.common.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:65)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFil ter.java:259)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :249)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)
2009.12.30 09:12:51 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: REM 18 FL 55a97183-9ec9-47d1-960a-d6555bd5c2c7 f530dfb6-9a55-483b-8bbc-cc04b2669407

java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)
2009.12.30 09:12:54 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: REM 19 FL 177032fc-a811-4c18-878d-c388d7041c91 f530dfb6-9a55-483b-8bbc-cc04b2669407

java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)
2009.12.30 09:13:26 [org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceMana gerImpl.java:271)
] Error storing offline presence of user:
java.sql.SQLException: Violation of PRIMARY KEY constraint ‘ofPresence_pk’. Cannot insert duplicate key in object ‘dbo.ofPresence’.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2816)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2254)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:631)
at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.execute(JtdsPreparedStatement.j ava:559)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.logicalcobwebs.proxool.ProxyStatement.invoke(ProxyStatement.java:100)
at org.logicalcobwebs.proxool.ProxyStatement.intercept(ProxyStatement.java:57)
at $java.sql.Wrapper$$EnhancerByProxool$$b2f44fdd.execute()
at org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceManag erImpl.java:268)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:145)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:112)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:176)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:134)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1133)
at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:202)
at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:185)
at org.jivesoftware.openfire.nio.ConnectionHandler.sessionClosed(ConnectionHandler .java:87)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.sessionClosed(A bstractIoFilterChain.java:550)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.common.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:65)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFil ter.java:259)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :249)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)
2009.12.30 09:15:06 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: ADC 14 FL C=86d9264b-67aa-468a-8c8d-d696cf3fa67a 5f088445-3fe7-43ca-a863-52b6990d659e

java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)

9

Would you mind loading a java-monitor plugin in your setup? That will give us more information, and will possibly allow you to anticipate a bit.

More information on java-monitor in this blogpost: New Openfire monitoring plugin

10

Alhough you’re writing that you’re using Java 1.6.0_17, your crashreport says otherwise:

Java HotSpot™ Server VM (1.6.0_03-b05) for windows-x86, built on Sep 24 2007 22:20:35 by “java_re” with unknown MS VC++:1310

03 is pretty old. It shouldn’t hurt to update.

11

Thanks for the quick reponse.

This is a crash report from when we where running 1.6.0_03-b05. We have updated since then thinking it may help solve our problems. We are running

1.6.0_17 Sun Microsystems Inc. – Java HotSpot™ Client VM.

I will look into Java-Monitor. In the mean time I have been running the JVMMonitor (http://www.igniterealtime.org/community/docs/DOC-1712). I have been making copies of the graphs directories when we have problems. I attached the graphs from when the system was running at 100% CPU and the admin console was not responding. (We get the logon page but a “Page can’t be displayed” upon logging in)
Graphs.zip (32373 Bytes)

12

have you tried running with out any plugins enabled.

13

No. The business will not let me remove the Krakken, Monitoring, Red5 or Search plugins. I could get away with removing the Client Control and Import/Export plugins…maybe… Unfortunately I have to work with what I have.

14

It just crashed again. This time the CPU didn’t peg to 100% but the service just stopped. I was in the console when it happened, memory was showing 78% of the 1280M for just over 900 users

A fatal error has been detected by the Java Runtime Environment:

java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

Internal Error (allocation.cpp:117), pid=1268, tid=1404

Error: ChunkPool::allocate

JRE version: 6.0_17-b04

Java VM: Java HotSpot™ Client VM (14.3-b01 mixed mode windows-x86 )

If you would like to submit a bug report, please visit:

http://java.sun.com/webapps/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread (0x56fd9800): JavaThread “CompilerThread0” daemon [_thread_in_native, id=1404, stack(0x57100000,0x57150000)]

Stack: [0x57100000,0x57150000], sp=0x5714f4b8, free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x1e66b7]
V [jvm.dll+0xa0c9c]
V [jvm.dll+0x27e6]
V [jvm.dll+0x29e2]
V [jvm.dll+0x2be3]
V [jvm.dll+0x52101]
V [jvm.dll+0x523b8]
V [jvm.dll+0x53ffc]
V [jvm.dll+0x54ef6]
V [jvm.dll+0x5513a]
V [jvm.dll+0x26a67]
V [jvm.dll+0x26b0a]
V [jvm.dll+0x26c71]
V [jvm.dll+0x26d84]
V [jvm.dll+0x84df0]
V [jvm.dll+0x858cf]
V [jvm.dll+0x1d0414]
V [jvm.dll+0x173e4c]
C [MSVCR71.dll+0x9565]
C [kernel32.dll+0x2482f]

Current CompileTask:
C1:4873 s! org.jivesoftware.util.log.output.io.rotate.RotatingFileTarget.write(Ljava/lang/ String;)V (46 bytes)

VM Arguments:
jvm_args: -Dexe4j.isInstall4j=true -Dexe4j.isService=true -Dexe4j.moduleName=C:\Program Files\Openfire\bin\openfire-service.exe -Dexe4j.processCommFile=C:\WINDOWS\TEMP\e4j_p1268.tmp -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Dexe4j.consoleCodepage=cp0 -Xrs -Xms1280m -Xmx1280m -XX:ThreadStackSize=128
java_command:
Launcher Type: generic

Environment Variables:
PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\The Open Group\WMI Mapper\bin;c:\program files\openfire\jre\bin
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel

--------------- S Y S T E M ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 2097151k(1158652k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot™ Client VM (14.3-b01) for windows-x86 JRE (1.6.0_17-b04), built on Oct 11 2009 00:52:06 by “java_re” with MS VC++ 7.1

time: Thu Dec 31 10:04:08 2009
elapsed time: 84400 seconds

-----------------------------------------------------------Last couple error.log errors------------------------------------------

2009.12.31 09:54:48 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)
]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at net.kano.joscar.flap.AsynchronousFlapProcessor.(AsynchronousFlapProcessor .java:29)
at net.kano.joscar.flap.ClientFlapConn.init(ClientFlapConn.java:82)
at net.kano.joscar.flap.ClientFlapConn.(ClientFlapConn.java:73)
at net.sf.kraken.protocols.oscar.AbstractFlapConnection.(AbstractFlapConnect ion.java:102)
at net.sf.kraken.protocols.oscar.LoginConnection.(LoginConnection.java:41)
at net.sf.kraken.protocols.oscar.OSCARSession.logIn(OSCARSession.java:120)
at net.sf.kraken.session.TransportSession.sessionDisconnected(TransportSession.jav a:523)
at net.sf.kraken.protocols.oscar.LoginConnection.handleStateChange(LoginConnection .java:65)
at net.sf.kraken.protocols.oscar.AbstractFlapConnection$1.stateChanged(AbstractFla pConnection.java:115)
at net.kano.joscar.net.ClientConn.setState(ClientConn.java:354)
at net.kano.joscar.net.ClientConn.connect(ClientConn.java:412)
at net.sf.kraken.protocols.oscar.OSCARSession.logIn(OSCARSession.java:124)
at net.sf.kraken.protocols.oscar.OSCARTransport.registrationLoggedIn(OSCARTranspor t.java:95)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:398)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:199)
at org.jivesoftware.openfire.component.InternalComponentManager$RoutableComponents .process(InternalComponentManager.java:619)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:260)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:164)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java: 337)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:176)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)
2009.12.31 09:55:39 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: PNG

java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)

15

That is unfortunate as any one of those plugins could be causing your error. I am fairly certain that it is being cuased by a plugin.

16

I hear ya…but my hands are tied…believe me, I have tried talking them into it.

17

seems rather rediculus to me that they would rather keep having it crash than try to diagnose the issue with some very basic tests.

18

The reference to swap space surprised me. I didn’t see that before. I’ve done a bit of googling, and found this:

3.1.4 Detail Message: request <size> bytes for <reason>. Out of swap space?

The detail message request <size> bytes for <reason>. Out of swap space? appears to be an OutOfMemoryError. However, the HotSpot VM code reports this apparent exception when an allocation from the native heap failed and the native heap might be close to exhaustion. The message indicates the size (in bytes) of the request that failed and the reason for the memory request. In most cases the <reason> part of the message is the name of a source module reporting the allocation failure, although in some cases it indicates a reason.

When this error message is thrown, the VM invokes the fatal error handling mechanism, that is, it generates a fatal error log file, which contains useful information about the thread, process, and system at the time of the crash. In the case of native heap exhaustion, the heap memory and memory map information in the log can be useful. See Appendix C, Fatal Error Log for detailed information about this file.

If this type of OutOfMemoryError is thrown, you might need to use troubleshooting utilities on the operating system to diagnose the issue further. See 2.16 Operating-System-Specific Tools.

The problem might not be related to the application, for example:

  • The operating system is configured with insufficient swap space.
  • Another process on the system is consuming all memory resources.

If neither of the above issues is the cause, then it is possible that the application failed due to a native leak, for example, if application or library code is continuously allocating memory but is not releasing it to the operating system.

I’ve found this at http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/memleaks.html#gbyvj

Did you rule out causes outside of Openfire?

19

It just crashed again today. I attached the JVM-Monitor graphs. These graphs are right after the service crashed but before rebooting or restarting the service. This server is dedicated to running Openfire. The only other software on this system is the antivirus software, which I disabled two weeks ago to exclude as a possible cause. This is a VMware server running on ESX so the VMtools are installed.

A fatal error has been detected by the Java Runtime Environment:

java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

Internal Error (allocation.cpp:117), pid=1256, tid=1412

Error: ChunkPool::allocate

JRE version: 6.0_17-b04

Java VM: Java HotSpot™ Client VM (14.3-b01 mixed mode windows-x86 )

If you would like to submit a bug report, please visit:

http://java.sun.com/webapps/bugreport/crash.jsp

VM Arguments:
jvm_args: -Dexe4j.isInstall4j=true -Dexe4j.isService=true -Dexe4j.moduleName=C:\Program Files\Openfire\bin\openfire-service.exe -Dexe4j.processCommFile=C:\WINDOWS\TEMP\e4j_p1256.tmp -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Dexe4j.consoleCodepage=cp0 -Xrs -Xms1280m -Xmx1280m -XX:ThreadStackSize=128
java_command:
Launcher Type: generic

Environment Variables:
PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\The Open Group\WMI Mapper\bin;c:\program files\openfire\jre\bin
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel

--------------- S Y S T E M ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 2097151k(1238272k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot™ Client VM (14.3-b01) for windows-x86 JRE (1.6.0_17-b04), built on Oct 11 2009 00:52:06 by “java_re” with MS VC++ 7.1

time: Mon Jan 04 11:38:05 2010
elapsed time: 253258 seconds
JVM.zip (178119 Bytes)

20

Since your hands are tied as to disabling the possible offiending software I do not know what help you possibly think we can provide.