Database connectivity problems? (3.2.2)

Setup:

  • Wildfire 3.2.2 and

  • MySQL 5.0.22 on

  • Linux (Fedora)

Symptoms:

  • Login problems:

  • First client login in the morning (Spark) never works. We always get an authentication error. Second try works. After first successful login no more login problems during the day (exiting spark, and then login - works perfectly).

  • First login to the web-interface: needs 3 attempts (always authentication error) until it works. Logout and then login again: no problems. (correction - I just tried again: even when it worked previously, it took 3 tries again now to successfully login)

  • Rooms: We have 2 chat rooms (one normal: A, one with restricted access only to certain users: B). A few days ago chat room B “vanished” (there’'s only A visible to the clients, and in the web-interface). But when I look into the database I can still see both chat rooms, A and B.

  • Lost group: We have only one group, where all users are included. Sometimes a user “loses” the group - i.e. he cannot see any other members as online in the client. And sometimes users vanish (even though they are still online - as can be verified in the web-interface).

In the error log I see the following exceptions:

2007.03.13 16:44:53 org.jivesoftware.wildfire.spi.PresenceManagerImpl.userAvailable(PresenceManagerI mpl.java:198)

java.sql.SQLException: No operations allowed after connection closed.Connection was implicitly closed due to underlying exception/error:

  • BEGIN NESTED EXCEPTION **

com.mysql.jdbc.CommunicationsException

MESSAGE: Communications link failure due to underlying exception:

  • BEGIN NESTED EXCEPTION **

java.net.SocketException

MESSAGE: Broken pipe

STACKTRACE:

java.net.SocketException: Broken pipe

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2739)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2650)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1581)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1695)

at com.mysql.jdbc.Connection.execSQL(Connection.java:3004)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1128)

at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1222)

at org.jivesoftware.database.ProfiledConnection$TimedPreparedStatement.executeQuer y(ProfiledConnection.java:780)

at org.jivesoftware.wildfire.group.DefaultGroupProvider.getGroupNames(DefaultGroup Provider.java:337)

at org.jivesoftware.wildfire.group.GroupManager.getGroups(GroupManager.java:343)

at org.jivesoftware.wildfire.roster.Roster.(Roster.java:91)

at org.jivesoftware.wildfire.roster.RosterManager.getRoster(RosterManager.java:94)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:255)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:100)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:88)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:151)

at org.jivesoftware.wildfire.PresenceRouter.handle(PresenceRouter.java:123)

at org.jivesoftware.wildfire.PresenceRouter.route(PresenceRouter.java:69)

at org.jivesoftware.wildfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:75)

at org.jivesoftware.wildfire.net.StanzaHandler.processPresence(StanzaHandler.java: 300)

at org.jivesoftware.wildfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:225)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:147)

at org.jivesoftware.wildfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:131)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:192)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :250)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:305)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

at java.lang.Thread.run(Thread.java:595)

  • END NESTED EXCEPTION **

Last packet sent to the server was 1 ms ago.

STACKTRACE:

com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:

  • BEGIN NESTED EXCEPTION **

java.net.SocketException

MESSAGE: Broken pipe

STACKTRACE:

java.net.SocketException: Broken pipe

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2739)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2650)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1581)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1695)

at com.mysql.jdbc.Connection.execSQL(Connection.java:3004)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1128)

at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1222)

at org.jivesoftware.database.ProfiledConnection$TimedPreparedStatement.executeQuer y(ProfiledConnection.java:780)

at org.jivesoftware.wildfire.group.DefaultGroupProvider.getGroupNames(DefaultGroup Provider.java:337)

at org.jivesoftware.wildfire.group.GroupManager.getGroups(GroupManager.java:343)

at org.jivesoftware.wildfire.roster.Roster.(Roster.java:91)

at org.jivesoftware.wildfire.roster.RosterManager.getRoster(RosterManager.java:94)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:255)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:100)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:88)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:151)

at org.jivesoftware.wildfire.PresenceRouter.handle(PresenceRouter.java:123)

at org.jivesoftware.wildfire.PresenceRouter.route(PresenceRouter.java:69)

at org.jivesoftware.wildfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:75)

at org.jivesoftware.wildfire.net.StanzaHandler.processPresence(StanzaHandler.java: 300)

at org.jivesoftware.wildfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:225)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:147)

at org.jivesoftware.wildfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:131)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:192)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :250)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:305)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

at java.lang.Thread.run(Thread.java:595)

  • END NESTED EXCEPTION **

Last packet sent to the server was 1 ms ago.

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2757)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2650)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1581)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1695)

at com.mysql.jdbc.Connection.execSQL(Connection.java:3004)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1128)

at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1222)

at org.jivesoftware.database.ProfiledConnection$TimedPreparedStatement.executeQuer y(ProfiledConnection.java:780)

at org.jivesoftware.wildfire.group.DefaultGroupProvider.getGroupNames(DefaultGroup Provider.java:337)

at org.jivesoftware.wildfire.group.GroupManager.getGroups(GroupManager.java:343)

at org.jivesoftware.wildfire.roster.Roster.(Roster.java:91)

at org.jivesoftware.wildfire.roster.RosterManager.getRoster(RosterManager.java:94)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:255)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:100)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:88)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:151)

at org.jivesoftware.wildfire.PresenceRouter.handle(PresenceRouter.java:123)

at org.jivesoftware.wildfire.PresenceRouter.route(PresenceRouter.java:69)

at org.jivesoftware.wildfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:75)

at org.jivesoftware.wildfire.net.StanzaHandler.processPresence(StanzaHandler.java: 300)

at org.jivesoftware.wildfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:225)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:147)

at org.jivesoftware.wildfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:131)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:192)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :250)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:305)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

at java.lang.Thread.run(Thread.java:595)

  • END NESTED EXCEPTION **

at com.mysql.jdbc.Connection.checkClosed(Connection.java:1844)

at com.mysql.jdbc.Connection.prepareStatement(Connection.java:4306)

at com.mysql.jdbc.Connection.prepareStatement(Connection.java:4272)

at org.jivesoftware.database.AbstractConnection.prepareStatement(AbstractConnectio n.java:36)

at org.jivesoftware.database.ProfiledConnection.prepareStatement(ProfiledConnectio n.java:608)

at org.jivesoftware.wildfire.spi.PresenceManagerImpl.userAvailable(PresenceManager Impl.java:193)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:110)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:88)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:151)

at org.jivesoftware.wildfire.PresenceRouter.handle(PresenceRouter.java:123)

at org.jivesoftware.wildfire.PresenceRouter.route(PresenceRouter.java:69)

at org.jivesoftware.wildfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:75)

at org.jivesoftware.wildfire.net.StanzaHandler.processPresence(StanzaHandler.java: 300)

at org.jivesoftware.wildfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:225)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:147)

at org.jivesoftware.wildfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:131)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:192)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :250)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:305)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

at java.lang.Thread.run(Thread.java:595)

Hi,

it seems that Wildfire did lose the connection to the database. Did you restart your database or do you have a firewall which closes open connections?

There is an open issue to improve the detection of closed connections.

LG

The database is running since quite a long time, no restart. Also there are several other applications which use MySQL at the same time with no problem.

Both Wildfire and MySQL run on the same machine, so I can’'t imagine firewall issues.

We seem to have the same issue, and it seems to be related to this post:

http://www.igniterealtime.org/forum/thread.jspa?messageID=141450&#141450

Something about sites with little traffic timing out the mysql connection but not realizing it, and failing.

We didn’'t use Wildfire very much before, so “little traffic” is true in our case.

Now we have upgraded to 3.2.4 and moved all our internal messaging to Wildfire, and this problem didn’'t occur anymore yet.