Eventually Jive / Openfire Loses All Client Connections (3.9.1)

Hi - We’re running 3.9.1 with JDK 1.6.0_33 on Solaris 11 with control of the service under SMF (a way to stop, start, etc. the service - which doesn’t work all that well, e.g. I usually need to kill -9 to stop, when disabling jive with SMF svcadm).

I’m not sure if using the included scripts would help in that matter, though I don’t think it’s related to the issue we’re having below.

We’re running using Oracle LDAP and Oracle database.

Jive works for a good while, maybe 20 to 30 or so days, then it seems to lose its client connections and no one can reconnect (until it is restarted).

I believe the admin web UI is available however. We have roughly 50-100 users active.

I can’t really correlate anything in the logs. I do see the following at a point I think may correlate:

2014.10.14 13:48:22 org.jivesoftware.openfire.net.SocketReadingMode - Error while negotiating TLS: org.jivesoftware.openfire.net.SocketConnection@2c423f16 socket: Socket[addr=/208.68.163.218,port=38572,localport=5269] session: org.jivesoftware.openfire.session.LocalIncomingServerSession@69aee1bd sta

tus: 1 address: xyz.edu/dadd7f67 id: dadd7f67

javax.net.ssl.SSLHandshakeException: Invalid padding

at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)

at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1528)

at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:932)

at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:845 )

at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)

at org.jivesoftware.openfire.net.TLSStreamHandler.doHandshake(TLSStreamHandler.jav a:222)

at org.jivesoftware.openfire.net.TLSStreamHandler.start(TLSStreamHandler.java:168)

at org.jivesoftware.openfire.net.SocketConnection.startTLS(SocketConnection.java:1 82)

at org.jivesoftware.openfire.net.SocketReadingMode.negotiateTLS(SocketReadingMode. java:85)

at org.jivesoftware.openfire.net.BlockingReadingMode.readStream(BlockingReadingMod e.java:138)

at org.jivesoftware.openfire.net.BlockingReadingMode.run(BlockingReadingMode.java: 76)

at org.jivesoftware.openfire.net.SocketReader.run(SocketReader.java:137)

at java.lang.Thread.run(Thread.java:662)

Caused by: javax.crypto.BadPaddingException: Padding length invalid: 216

at com.sun.net.ssl.internal.ssl.CipherBox.removePadding(CipherBox.java:405)

at com.sun.net.ssl.internal.ssl.CipherBox.decrypt(CipherBox.java:253)

at com.sun.net.ssl.internal.ssl.InputRecord.decrypt(InputRecord.java:153)

at com.sun.net.ssl.internal.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:2 38)

at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:914)

… 11 more

2014.10.14 14:00:48 org.jivesoftware.openfire.nio.ConnectionHandler - ConnectionHandler reports IOException for session: (SOCKET, R: /172.29.1.228:54798, L: /128.122.108.167:5223, S: 0.0.0.0/0.0.0.0:5223)

java.io.IOException: Connection reset by peer

at sun.nio.ch.FileDispatcher.read0(Native Method)

at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)

at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)

at sun.nio.ch.IOUtil.read(IOUtil.java:171)

at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.j ava:218)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcesso r.java:198)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProce ssor.java:45)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:485)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

I also see quite a lot of this (but think I know what it is …our DNS doesn’t point to Google for chat, but rather to our openfire instance, though we do use google apps, and have chat enabled, and use it w/in the google apps context)…

2014.10.15 10:53:29 org.jivesoftware.openfire.server.ServerDialback - ServerDialback: OS - Ignoring unexpected answer in validation from: gmail.com id: E156367E16B84DD5 for domain: xyz.edu answer:<stream:error xmlns:stream=“http://etherx.jabber.org/streams”><str:text xmlns:str=“urn:ietf:params:xml:ns:xmpp-streams”>xyz.edu is a Google Apps Domain with Talk service enabled.</str:text></stream:error>

root@jive:/apps/openfire/logs#