Openfire kills itselfs and doesn't start today!? (OutOfMemoryError + no connection to embedded DB)

Jabber-Server.de · January 6, 2010, 12:57pm

Last night openfire crasged. From what i figured out so far because of an GC overhead limit exceeded.

This is what warn.log gives me:

2010.01.06 05:53:01 Problem scavenging sessions java.lang.OutOfMemoryError: GC overhead limit exceeded
2010.01.06 09:43:36 No ACK was received when sending stanza to: org.jivesoftware.openfire.nio.NIOConnection@1001894 MINA Session: (SOCKET, R: /95.33.124.193:49172, L: /92.51.129.253:5223, S: 0.0.0.0/0.0.0.0:5223)
2010.01.06 10:08:54 Shutdown waited for 0 milliseconds for all the connections to become inactive but the active count is still 1. Shutting down anyway.

The error.log:

2010.01.06 09:16:02 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)] Couldn’t get connection
java.lang.NullPointerException

Stdout.log tells me: There was a error starting the server. See logfiles for further informations.

Stderror.log:

Exception in thread “HSQLDB Timer @1aa9f99” java.lang.OutOfMemoryError: Java heap space
Database setup or configuration error: Please verify your database settings and check the logs/error.log file for detailed error messages.
Logging Error: Unknown error writing event.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Database could not be accessed
java.sql.SQLException: ConnectionManager.getConnection() failed to obtain a connection after 11 retries. The exception from the last attempt is as follows: java.sql.SQLException: General error: java.lang.OutOfMemoryError: Java heap space
at org.jivesoftware.database.DbConnectionManager.getConnection(DbConnectionManager .java:124)
at org.jivesoftware.openfire.XMPPServer.verifyDataSource(XMPPServer.java:698)
at org.jivesoftware.openfire.XMPPServer.start(XMPPServer.java:427)
at org.jivesoftware.openfire.XMPPServer.(XMPPServer.java:161)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessor Impl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor AccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:106)
at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja va:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
at com.install4j.runtime.Launcher.main(Unknown Source)

info.log:

2010.01.06 07:38:08 Unable to get a connection from the database pool (attempt 3 out of 10).
java.sql.SQLException: java.lang.NullPointerException

Thats what i got so far. If i try to start openfire it seems to be sarting. “The deamon is running” i get as result. But after some seconds the deamon stopped without notice.

The deamon is stopped but my system process shows up:

PID
%CPU
%MEM
Command
Nice
Pri
RSS
Stat
Time
User

30660
99.9
2.7
/usr/bin/java -server -Dinstall4j.jvmDir=/usr -Dexe4j.moduleName=/opt/openfire/bin/openfire -classpath /opt/openfire/.install4j/i4jruntime.jar:/opt/openfire/lib/activation.jar:/opt/o penfire/lib/bouncycastle.jar:/opt/openfire/lib/commons-el.jar:/opt/openfi
0
21
921952
S
00:07:16
10015

Now i’m stuck. I can’t start the server and i don’t no why. So i’m looking for some help

Jabber-Server.de · January 6, 2010, 3:05pm

In folder /embedded-db/ file openfire.properties says:

#HSQL Database Engine
#Wed Jan 06 10:08:54 CET 2010
hsqldb.script_format=0
runtime.gc_interval=0
sql.enforce_strict_size=false
hsqldb.cache_size_scale=8
readonly=false
hsqldb.nio_data_file=true
hsqldb.cache_scale=14
version=1.8.0
hsqldb.default_table_type=memory
hsqldb.cache_file_scale=1
hsqldb.log_size=200
modified=yes
hsqldb.cache_version=1.7.0
hsqldb.original_version=1.8.0
hsqldb.compatible_version=1.8.0

Is here any hind given why openfire can’t connect to the db?

Guenther_Niess · January 6, 2010, 4:00pm

Hi,

rene-1 wrote:

In folder /embedded-db/ file openfire.properties says:

…

Is here any hind given why openfire can’t connect to the db?

No, what files are in the embedded-db folder? Could you try to start the embedded db viewer with:

java -cp lib/hsqldb.jar org.hsqldb.util.DatabaseManagerSwing --url jdbc:hsqldb:./embedded-db/openfire --user sa --password pw

Does it start correctly and can it access the embedded db?

Jabber-Server.de · January 6, 2010, 4:23pm

Files within the embedded-db folder are:

openfire.properties
openfire.log (empty)
openfire.script (~ 785 MB)

What URL should i type in at java -cp lib/hsqldb.jar org.hsqldb.util.DatabaseManagerSwing --url ??

And username and password are the same as login into the admin console?

LG1 · January 6, 2010, 6:25pm

Hi,

increase the Xmx value of Openfire, this should help. I wonder why your openfire.script is 1 GB, do you archive logs or something like this?

LG

Jabber-Server.de · January 7, 2010, 10:19am

I’ve increased the xmx value and also copied the original files from the 3.6.4 release tar and replaced them with my custom conf, plugins … like an normal update. However now i’m able to start openfire and get to the setup screen. I tried walk through but when it comes to db selection an i choose the embedded he reloads itselfs for a long time and end up crashing with the following message:

HTTP ERROR: 500

Java heap space

RequestURI=/setup/setup-datasource-settings.jsp

Caused by:

java.lang.OutOfMemoryError: Java heap space

And yeah, my openfire.script is this big cause i forgot to turn archive off

Jabber-Server.de · January 7, 2010, 12:09pm

Finally i was able to complete the setup with an xmx value of 2560m! But thats quiet a lot an i think my machine can’t take this for long. Within the admin console i see 1918.12 MB of 2465.19 MB (77.8%) used - and its still raisin’ slowly.

I’ve started openfire with a bunch of parameters:

-Djava.net.preferIPv4Stack=true -Xms768m -Xmx2560m -Xss128k -Xoss128k -XX:ThreadStackSize=128 -XX:+PrintGCDetails -Xloggc:/opt/openfire/bin/gc.log -XX:+HeapDumpOnOutOfMemoryError"

Is this a wise decision?

Guenther_Niess · January 7, 2010, 2:01pm

I would recommend to migrate to another DBMS such as MySQL or cleanup your database.

Jabber-Server.de · January 7, 2010, 2:06pm

Sure, this would be good. But how to migrate from the embedeed db to mysql or to clean up the embedded db?

LG1 · January 7, 2010, 5:02pm

Hi,

get some unixtools and run something like cat openfire.script | grep -v ARCHIVE >openfire-mini.script´. "grep -v ARCHIVE" expression will drop every line which contains "ARCHIVE", so you'll likely get a very small Openfire database. Withcat openfire.script | grep -v ARCHIVE >openfire-archive.txt´ you’ll get every line which contains “ARCHIVE” in another file.

You can do this while Openfire is running, then review the “openfire-archive.txt” file and make sure that it contains only chat archives and not rosters which are named archive or something like this. Then rename the files openfire.script to openfire.script.huge and openfire-mini.script to openfire.script and start Openfire again.

LG

LG1 · January 7, 2010, 5:03pm

Migration from embedded-db to MySQL, Oracle, MSSQL, …

http://www.igniterealtime.org/community/docs/DOC-1540