powered by Jive Software

Openfire kills itselfs and doesn't start today!? (OutOfMemoryError + no connection to embedded DB)

Last night openfire crasged. From what i figured out so far because of an GC overhead limit exceeded.

This is what warn.log gives me:

2010.01.06 05:53:01 Problem scavenging sessions java.lang.OutOfMemoryError: GC overhead limit exceeded
2010.01.06 09:43:36 No ACK was received when sending stanza to: org.jivesoftware.openfire.nio.NIOConnection@1001894 MINA Session: (SOCKET, R: /95.33.124.193:49172, L: /92.51.129.253:5223, S: 0.0.0.0/0.0.0.0:5223)
2010.01.06 10:08:54 Shutdown waited for 0 milliseconds for all the connections to become inactive but the active count is still 1. Shutting down anyway.

The error.log:

2010.01.06 09:16:02 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)] Couldn’t get connection
java.lang.NullPointerException

Stdout.log tells me: There was a error starting the server. See logfiles for further informations.

Stderror.log:

Exception in thread “HSQLDB Timer @1aa9f99” java.lang.OutOfMemoryError: Java heap space
Database setup or configuration error: Please verify your database settings and check the logs/error.log file for detailed error messages.
Logging Error: Unknown error writing event.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Database could not be accessed
java.sql.SQLException: ConnectionManager.getConnection() failed to obtain a connection after 11 retries. The exception from the last attempt is as follows: java.sql.SQLException: General error: java.lang.OutOfMemoryError: Java heap space
at org.jivesoftware.database.DbConnectionManager.getConnection(DbConnectionManager .java:124)
at org.jivesoftware.openfire.XMPPServer.verifyDataSource(XMPPServer.java:698)
at org.jivesoftware.openfire.XMPPServer.start(XMPPServer.java:427)
at org.jivesoftware.openfire.XMPPServer.(XMPPServer.java:161)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessor Impl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor AccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:106)
at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja va:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
at com.install4j.runtime.Launcher.main(Unknown Source)

info.log:

2010.01.06 07:38:08 Unable to get a connection from the database pool (attempt 3 out of 10).
java.sql.SQLException: java.lang.NullPointerException

Thats what i got so far. If i try to start openfire it seems to be sarting. “The deamon is running” i get as result. But after some seconds the deamon stopped without notice.

The deamon is stopped but my system process shows up:

PID
%CPU
%MEM
Command
Nice
Pri
RSS
Stat
Time
User

30660
99.9
2.7
/usr/bin/java -server -Dinstall4j.jvmDir=/usr -Dexe4j.moduleName=/opt/openfire/bin/openfire -classpath /opt/openfire/.install4j/i4jruntime.jar:/opt/openfire/lib/activation.jar:/opt/o penfire/lib/bouncycastle.jar:/opt/openfire/lib/commons-el.jar:/opt/openfi
0
21
921952
S
00:07:16
10015

Now i’m stuck. I can’t start the server and i don’t no why. So i’m looking for some help :slight_smile:

In folder /embedded-db/ file openfire.properties says:

#HSQL Database Engine
#Wed Jan 06 10:08:54 CET 2010
hsqldb.script_format=0
runtime.gc_interval=0
sql.enforce_strict_size=false
hsqldb.cache_size_scale=8
readonly=false
hsqldb.nio_data_file=true
hsqldb.cache_scale=14
version=1.8.0
hsqldb.default_table_type=memory
hsqldb.cache_file_scale=1
hsqldb.log_size=200
modified=yes
hsqldb.cache_version=1.7.0
hsqldb.original_version=1.8.0
hsqldb.compatible_version=1.8.0

Is here any hind given why openfire can’t connect to the db?

Hi,

rene-1 wrote:

In folder /embedded-db/ file openfire.properties says:

Is here any hind given why openfire can’t connect to the db?

No, what files are in the embedded-db folder? Could you try to start the embedded db viewer with:

java -cp lib/hsqldb.jar org.hsqldb.util.DatabaseManagerSwing --url jdbc:hsqldb:./embedded-db/openfire --user sa --password pw

Does it start correctly and can it access the embedded db?

Files within the embedded-db folder are:

  • openfire.properties

  • openfire.log (empty)

  • openfire.script (~ 785 MB)

What URL should i type in at java -cp lib/hsqldb.jar org.hsqldb.util.DatabaseManagerSwing --url ??

And username and password are the same as login into the admin console?

Hi,

increase the Xmx value of Openfire, this should help. I wonder why your openfire.script is 1 GB, do you archive logs or something like this?

LG

I’ve increased the xmx value and also copied the original files from the 3.6.4 release tar and replaced them with my custom conf, plugins … like an normal update. However now i’m able to start openfire and get to the setup screen. I tried walk through but when it comes to db selection an i choose the embedded he reloads itselfs for a long time and end up crashing with the following message:

HTTP ERROR: 500

Java heap space

RequestURI=/setup/setup-datasource-settings.jsp

Caused by:

java.lang.OutOfMemoryError: Java heap space

And yeah, my openfire.script is this big cause i forgot to turn archive off

Finally i was able to complete the setup with an xmx value of 2560m! But thats quiet a lot an i think my machine can’t take this for long. Within the admin console i see 1918.12 MB of 2465.19 MB (77.8%) used - and its still raisin’ slowly.

I’ve started openfire with a bunch of parameters:

-Djava.net.preferIPv4Stack=true -Xms768m -Xmx2560m -Xss128k -Xoss128k -XX:ThreadStackSize=128 -XX:+PrintGCDetails -Xloggc:/opt/openfire/bin/gc.log -XX:+HeapDumpOnOutOfMemoryError"

Is this a wise decision?

I would recommend to migrate to another DBMS such as MySQL or cleanup your database.

Sure, this would be good. But how to migrate from the embedeed db to mysql or to clean up the embedded db?

Hi,

get some unixtools and run something like cat openfire.script | grep -v ARCHIVE >openfire-mini.script´. "grep -v ARCHIVE" expression will drop every line which contains "ARCHIVE", so you'll likely get a very small Openfire database. Withcat openfire.script | grep -v ARCHIVE >openfire-archive.txt´ you’ll get every line which contains “ARCHIVE” in another file.

You can do this while Openfire is running, then review the “openfire-archive.txt” file and make sure that it contains only chat archives and not rosters which are named archive or something like this. Then rename the files openfire.script to openfire.script.huge and openfire-mini.script to openfire.script and start Openfire again.

LG

Migration from embedded-db to MySQL, Oracle, MSSQL, …

http://www.igniterealtime.org/community/docs/DOC-1540