SSL handshake error

I tried to apply openfire in production and faced with one problem. Here is my investigation:

One of 50 computers can’t connect to openfire (any user on this computer)

Miranda NG writed " SSL handshake error 80090304 381…", Pandion writed nothing but could not connect. And Spark connected normal.

The error in Openfire warning log was:

2015.11.16 16:31:59 org.jivesoftware.openfire.nio.ConnectionHandler - Closing connection due to exception in session: (0x00000180: nio socket, server, /192.168.1.4:54518 => /192.168.0.249:5222)
javax.net.ssl.SSLHandshakeException: SSL handshake failed.
        at org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:487)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
        at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:410)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:710)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:664)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:653)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1124)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLException: Unsupported record version Unknown-47.115
        at sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:116)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:854)
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:767)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at org.apache.mina.filter.ssl.SslHandler.unwrap(SslHandler.java:728)
        at org.apache.mina.filter.ssl.SslHandler.unwrapHandshake(SslHandler.java:666)
        at org.apache.mina.filter.ssl.SslHandler.handshake(SslHandler.java:552)
        at org.apache.mina.filter.ssl.SslHandler.messageReceived(SslHandler.java:351)
        at org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:468)
        ... 15 more

Miranda NG (and may be Pandion) uses system schanell.dll to made ssl connections and Spark use java do that.

I have not found any solutions of this error and tried debug connection with Wireshark:

Here is screenshots (bad connection on the left, good on the right)

The differences are marked. The problem computer has installed CryptoPro (Russian crypto program that added GOST algorithm and some other features, but i have 5 other computers with CryptoPro installed and they work normal)

Here is server answers:

Openfire answered with different certificates.

Here is the rest of bad dialog:

Miranda NG closes the stream but openfire, as i understand, waits something other and send fatal error.

There is no any setting that rearranges cipher suits on bad computer, and I don’t know why TLS_DHE_DSS_WITH_AES_128_CBC_SHA became the second cipher. I reordered cipher suits using group policy to default and Miranda could connect.

I’ve solved problem for me and it may be help to solve [OF-793] javax.net.ssl.SSLException: Unsupported record version Unknown-47.115 - Jive Software Open Source

Here is my conclusions and questions:

  1. Default windows SSL engine does not understand Openfire DSA self-signed certificate. Is it problem of openfire or java? or even windows?

  2. Is Openfire should send fatal error or should change cryptoalgorithm? It is normal reaction on “< /stream:stream>” ?

  3. Is there any instructions how generate and import certificates from Microsoft CA to Openfire? May be certificate generated by Microsoft Enterprise CA will be more understandable by Windows SCHANELL.

I’m not a big ssl guru. Sorry if my questions are silly.

I’ve managed to sign my both cetrificates by Enterprise Microsoft CA and it does not help. I still could not connect with TLS_DHE_DSS_WITH_AES_128_CBC_SHA.

Now I think that problem is not in Certificate.

PS for CriptoPro users: to bring normal cipher suites order back you need to set this checkbox Кабинет налогоплательщика