CPU spikes with 1.6u39/1.7u13 JRE and Hazelcast

Openfire Openfire Support
1

I’ve been running OF 3.7.1 with 1.7u5 for a long time in a two node configuration with Coherence for clustering. I recently upgraded our DR environment to 3.8.1 from SVN and replaced Coherence with Hazelcast. With 1.7u5 it works great - It was running for 36 hours with 2 users connected and CPU was basically flat.

With either of the current JREs 1.6u39 or 1.7u5 I see periodic high CPU on both nodes. Typically the CPU utilization lasts for a minute or so, however it corresponds with an exception in org.jivesoftware.openfire.nio.ConnectionHandler on the other node. Both nodes seem to periodically experience this exception due to SSL padding issues from health checks from our load balancers (which is really just XMPPHP connecting and authenticating using TLS). Even just running for a couple of hours I see spikes up to 10% CPU.

I’ve seen lots of people complaining about SSL padding errors on Java 1.7, but never any real solutions. Has anyone experienced similar performance anolomies with Openfire and Java 1.7?

2

Hi, Please verify that you are running the trunk version of hazelcast as well? daryl

3

Daryl-

Openfire says 1.0.4 - I’ll do a fresh checkout of trunk and compile it to make sure I am current.

4

I’m now running rev 13498 - One exception is I am still running 1.2.0 of the monitoring plugin. All the other plugins are copied out of the build however.

5

To better illustrate this issue here is CPU graph for one system over the last week. 15th/16th we were running JRE 1.7u5. Around 9am on 17th I switched it to 1.7u13.

6

For what it is worth, I’m having the same issue with the OpenJDK implementation of Java 1.7. Testing OpenJDK 1.6 now.

Does anyone know why we are seeing SSL Padding errors with more recent JREs? Looks like some other projects had to code around this, or at least change how buffering works with SSL connections.

7

Flow made some notes about this on ticket: OF-636

8

Seems like a different issue - I just get SSL Padding Errors with any current 1.6/1.7 JRE (either Sun/Oracle or OpenJDK)

013.02.28 08:49:21 org.jivesoftware.openfire.nio.ConnectionHandler - ConnectionHandler reports IOException for session: (SOCKET, R: /169.254.202.1:44753, L: /169.254.202.2:5222, S: 0.0.0.0/0.0.0.0:5222)

javax.net.ssl.SSLHandshakeException: SSL handshake failed.

at org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java:416)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.support.AbstractIoFilterChain$HeadFilter.messageReceived (AbstractIoFilterChain.java:499)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.fireMessageReceived(Abstra ctIoFilterChain.java:293)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.j ava:228)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcesso r.java:198)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProce ssor.java:45)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:485)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: javax.net.ssl.SSLHandshakeException: Invalid padding

at sun.security.ssl.Alerts.getSSLException(Unknown Source)

at sun.security.ssl.SSLEngineImpl.fatal(Unknown Source)

at sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)

at sun.security.ssl.SSLEngineImpl.readNetRecord(Unknown Source)

at sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)

at javax.net.ssl.SSLEngine.unwrap(Unknown Source)

at org.apache.mina.filter.support.SSLHandler.unwrap0(SSLHandler.java:658)

at org.apache.mina.filter.support.SSLHandler.unwrapHandshake(SSLHandler.java:614)

at org.apache.mina.filter.support.SSLHandler.handshake(SSLHandler.java:493)

at org.apache.mina.filter.support.SSLHandler.messageReceived(SSLHandler.java:306)

at org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java:392)

… 14 more

Caused by: javax.crypto.BadPaddingException: Padding length invalid: 132

at sun.security.ssl.CipherBox.removePadding(Unknown Source)

at sun.security.ssl.CipherBox.decrypt(Unknown Source)

at sun.security.ssl.InputRecord.decrypt(Unknown Source)

at sun.security.ssl.EngineInputRecord.decrypt(Unknown Source)

… 23 more