MySQL connection pool problems in 4.1.1

Jindrich_Sarson · January 3, 2017, 3:52pm

Hi,

I just upgraded to 4.1.1 and immediately after startup the problems with MySQL connection pool begins. All 50 connections in pool are used according to Connection pool info in few seconds and 0 available connections became permanent. This results into unavailability to connect by any user.

When downgrade to 4.0.4, everything works fine.

Tried with both Java 7 and Java 8.

Logs:

2017.01.03 16:39:06 org.jivesoftware.openfire.auth.DefaultAuthProvider - User SQL failure:

java.sql.SQLException: ConnectionManager.getConnection() failed to obtain a connection after 11 retries. The exception from the last attempt is as follows: java.sql.SQLException: Couldn’t get connection because we are at maximum connection count (50/50) and there are none available

at org.jivesoftware.database.DbConnectionManager.getConnection(DbConnectionManager .java:156)

at org.jivesoftware.openfire.auth.DefaultAuthProvider.getUserInfo(DefaultAuthProvi der.java:89)

at org.jivesoftware.openfire.auth.DefaultAuthProvider.getUserInfo(DefaultAuthProvi der.java:119)

Any idea how to fix this?

Thanks,

Jindrich

guus · January 3, 2017, 7:03pm

Hmm, that’s not good. Can you please create a Java thread dump of the Openfire process when this happens again? That will show us what code is keeping those connections busy.

Anything in the MySQL slow log? Does SHOW PROCESSLIST give any indication of what is going on?

Cristian4 · January 3, 2017, 9:15pm

I had the same problem and i could fixed just into the openfire database put:

mysql> GRANT ALL on openfire.* to ‘openfire’@localhost IDENTIFIED BY ‘secretpassword’;

mysql> FLUSH PRIVILEGES;

akrherz · January 4, 2017, 2:30am

Cristian,

I don’t think your issue is related, but was this a new install?

Jindrich_Sarson · January 4, 2017, 1:06pm

Hi,

unfortunately, there is nothing in mysql slow log.

I attached DB connections details page from OF.

Generally, when Openfire starts and I send medium load there, this DB connection not available happens immediately.

On the second server, another instance but 3.9.3 is running agains the same database (and it works). I am trying to switch to this new one - so that when I shutdown 3.9.3 and startup 4.1.1 and transfer traffic there, this problem happens.

I also tried to get dump as you requested, but after half an hour it is still processing. Shouldn’t take so long, right? I am using:

jmap -F -dump:format=b,file=openfire.hprof 1384
Openfire Admin Console: Database Properties.pdf (78862 Bytes)

guus · January 4, 2017, 5:45pm

jmap will generate a heap dump (all memory, which can be pretty big). I asked only for a thread dump (a stack trace of each thread in Openfire, couple of kilobytes of text). You’d typically generate one by sending a QUIT signal to a Java process

guus · January 23, 2017, 7:44pm

@Daryl Herzmann created an issue for this problem in our bugtracker: [OF-1278] Recursive Loop in SCRAM salt generation - IgniteRealtime JIRA