Sending messages to a bunch of users takes pretty long

Hey,

I’m having an issue with my Openfire server sending messages to like 1000 users at once. On my old server it took like 15-30 seconds to send a message via the broadcast plugin to all my users. Now, on the new server it takes like 10 minutes.

The thing is, it’s probably not the brodcast plugin being the issue. I wrote a script that connects to Openfire to send the message to all users. Sending the message actually takes less then a second, but it takes a lot of time for people to receive the message. Also sent a message to all online users only. At the time of testing, I had like 150 users being online. Even that took quite some time. Not 10 minutes, but still around 1-2 minutes.

The big difference between the old and the new server ist he CPU. Old server had an Intel Xeon dualcore with 3 GHz/core. The new server has an Intel i3 Quadcore with 1.6 Ghz/core.

So, what’s the problem here?

Things I’ve already tried:

  • deleted all offline messages in the DB
  • disabled debug logging to reduce i/o
  • switched from OpenJDK to Oracle JDK

That is really odd. Cam you create a couple of thread dumps while Openfire is busy doing the broadcast?

@Guus der Kinderen I hope I understood your reply right and created a few dumps with jstack pre, while and post broadcast.
dumps.zip (43234 Bytes)

Thanks, that was helpful.

It appears that your server is spending quite some time storing messages for those users that are not online at the time when you send the broadcast. This can have a number of causes:

  • The Openfire plugin might not be very efficient.
  • You might have a large amount of (offline) users.
  • You might not want to send a broadcast to offline users in the first place.
  • Your database server is not operating efficiently (hardware might not be strong enough, indexing might be inefficient, etc).

“socket_c2s-thread-8” #63 daemon prio=5 os_prio=0 tid=0x0000638da0004800 nid=0x6598 runnable [0x0000638d56f06000]

java.lang.Thread.State: RUNNABLE

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

at java.net.SocketInputStream.read(SocketInputStream.java:170)

at java.net.SocketInputStream.read(SocketInputStream.java:141)

at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)

at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(Re adAheadInputStream.java:159)

at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)

- locked <0x0000638e1ba64b08> (a com.mysql.jdbc.util.ReadAheadInputStream)

at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)

at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3615)

at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4155)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)

at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)

- locked <0x0000638e1ba64b58> (a com.mysql.jdbc.JDBC4Connection)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)

- locked <0x0000638e1ba64b58> (a com.mysql.jdbc.JDBC4Connection)

at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)

- locked <0x0000638e1ba64b58> (a com.mysql.jdbc.JDBC4Connection)

at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)

- locked <0x0000638e1ba64b58> (a com.mysql.jdbc.JDBC4Connection)

at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)

at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) ...

In all ‘while…’ dumps. It seems that MySQL has a problem processing a packet. You should enable the mysql query log and check the response times there. I don’t think that we log this in Openfire.

Already talked to Guus on Jabber. He already told me, that the MySQL DB is the problem. I did some tweaks and got the time it takes to broadcast to all users down to like 1 minute. Also removed old offline messages and reduced the quota to 100 kb. Looks like most of that helped a lot. Also realized that all messages have been logged in the database as well which is why I disabled the plugin completely.

Out of curiosity, I enabled the query log. Every insert into ofOffiline has a time value between 170-200.

0,2s * 1000 users ~ 3 minutes. This explains why Openfire took so long and one should indeed look at the MySQL database.

I think you’re right. Eventhough it got a bit faster than tha in the meantime, but I think it’s just the hardware being to weak to handle it faster.