Certificate issues preventing connections to jabber.org

Nicholas_Killewald · March 23, 2016, 5:48pm

After upgrading to Openfire 4.0.2, I’ve run into a problem connecting to other servers (namely, jabber.org). Though I can log in to my own server, it’s not making any outside connections. Looking through the info logs, I ran across this (my own server info scrubbed):

2016.03.23 13:00:43 org.jivesoftware.openfire.session.LocalOutgoingServerSession[Create outgoing session for: (my_server) to jabber.org] - STARTTLS negotiation failed. Closing connection (without sending any data such as or ).

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)

    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)

    at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)

    at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)

    at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)

    at org.jivesoftware.openfire.net.TLSStreamHandler.doHandshake(TLSStreamHandler.jav a:293)

    at org.jivesoftware.openfire.net.TLSStreamHandler.start(TLSStreamHandler.java:178)

    at org.jivesoftware.openfire.net.SocketConnection.startTLS(SocketConnection.java:1 95)

    at org.jivesoftware.openfire.session.LocalOutgoingServerSession.secureAndAuthentic ate(LocalOutgoingServerSession.java:407)

    at org.jivesoftware.openfire.session.LocalOutgoingServerSession.createOutgoingSess ion(LocalOutgoingServerSession.java:305)

    at org.jivesoftware.openfire.session.LocalOutgoingServerSession.authenticateDomain (LocalOutgoingServerSession.java:210)

    at org.jivesoftware.openfire.server.OutgoingSessionPromise$PacketsProcessor.sendPa cket(OutgoingSessionPromise.java:267)

    at org.jivesoftware.openfire.server.OutgoingSessionPromise$PacketsProcessor.run(Ou tgoingSessionPromise.java:243)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at java.lang.Thread.run(Thread.java:745)

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)

    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)

    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)

    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)

    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)

    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)

    at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)

    at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)

    at java.security.AccessController.doPrivileged(Native Method)

    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)

    at org.jivesoftware.openfire.net.TLSStreamHandler.doTasks(TLSStreamHandler.java:34 4)

    at org.jivesoftware.openfire.net.TLSStreamHandler.doHandshake(TLSStreamHandler.jav a:254)

    ... 10 more

Caused by: java.security.cert.CertificateException: The provided input should not contain multiple certificates with identical subjectDN values. Offending value: CN=Let’s Encrypt Authority X1, O=Let’s Encrypt, C=US

    at org.jivesoftware.openfire.keystore.CertificateUtils.order(CertificateUtils.java :178)

    at org.jivesoftware.openfire.keystore.CertificateUtils.identifyEndEntityCertificat e(CertificateUtils.java:234)

    at org.jivesoftware.openfire.keystore.OpenfireX509TrustManager.checkServerTrusted( OpenfireX509TrustManager.java:99)

    at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl. java:936)

    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)

    ... 18 more

Now, while I DO have a Let’s Encrypt cert elsewhere on the system, that’s not what my Openfire install is using. That’s just using a plain self-signed cert created via the first-time setup when I first installed it (this Openfire setup is just a personal-use one, more or less for fun). In fact, I wasn’t aware I had a Let’s Encrypt authority sitting around in the first place. But, double-checking the Openfire code in GitHub, sure enough, that’s what’s throwing the exception.

I’m afraid I don’t have quite enough experience in managing SSL certs and/or authorities to figure out why Openfire thinks there’s multiple certs with identical subjectDNs, but unless the Let’s Encrypt installer did something weird, I’m fairly certain I haven’t done anything SSL-related above and beyond whatever Ubuntu Server usually does. Can anyone help?

This is a 64-bit Ubuntu 14.04.4 LTS system, running OpenJDK 8. Output of java -version follows:

openjdk version “1.8.0_72-internal”

OpenJDK Runtime Environment (build 1.8.0_72-internal-b15)

OpenJDK 64-Bit Server VM (build 25.72-b15, mixed mode)

Michael42 · March 23, 2016, 8:25pm

My Openfire servers are also experiencing this same issue connecting with jabber.org, except in my case, I’m still on Openfire version 4.0.1. The problem all of a sudden began yesterday 3/22 - I think it has to do with the possibility that jabber.org decided to switch their CA root from “StartCom” to “Let’s Encrypt”. As a result, the Openfire servers are only receiving incoming communication from jabber.org while the outgoing communication to jabber.org servers fail authentication during STARTTLS negotiation.

It seems that it is no longer enough just to import the CA Root as it was in the past. I tried importing the “Let’s Encrypt” root certificate as well as various intermediate certificates into Openfire but that did not solve the situation. The logs display:

"STARTTLS negotiation failed. Closing connection (without sending any data such as or ). "

as well as:

"Caused by: java.security.cert.CertificateException: The provided input should not contain multiple certificates with identical subjectDN values. Offending value: CN=Let’s Encrypt Authority X1, O=Let’s Encrypt, C=US "

You can see the IM Observatory S2S test results for jabber.org here:

https://xmpp.net/result.php?domain=jabber.org&type=server

At this point, I’m not certain if the problem is with the new certificate or whether it is with how the Openfire code reads this particular certificate or chain. I emailed Peter Saint-Andre at jabber.org yesterday and I’m awaiting his reply. If I find a solution, I will post it on this forum.

–Michael

Nicholas_Killewald · March 23, 2016, 8:34pm

Huh. Well, I’ll admit that jabber.org was the only server I was testing against, so it’s possible it’s just something amiss on their end. If it is, that’s one really weird coincidence with the timing of my upgrade.

I guess we’ll just wait for Peter’s response, then. Thanks!

guus · March 25, 2016, 9:59am

As you noted correctly, the jabber.org domain quite recently switched to Let’s Encrypt.

Openssl gives us the current certificate chain from jabber.org:

$ openssl s_client -connect jabber.org:5222 </dev/null -starttls xmpp
(snip)
---
Certificate chain
0 s:/CN=jabber.org
  i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
  i:/O=Digital Signature Trust Co./CN=DST Root CA X3
2 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
  i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
---
(snip)

There’s a compound problem here:

The Let’s Encrypt CA certificates do currently not ship with Openfire. Even when they did, those would only be available in new Openfire installs, not updates. For existing installs, administrators will have to manually import the Let’s Encrypt CA certificates in the Openfire truststore(s). You’ve already found this.
The certificate chain that is served by jabber.org is not a linear chain. Instead, it’s a tree structure, where the same end entity certificate appears to be validated by two distinct root certificates. I assume that this is done for compatibility sake.
Openfire tries to order the certificate chain that it receives, but assumes a linear chain. It fails on the tree-like structure that’s now in the jabber.org chain.

At this time, I’ve yet to find the best solution for this. Some considerations:

Even if the TLS RFC defines such a chain to be invalid (“This is a sequence (chain) of certificates. The sender’s certificate MUST come first in the list. Each following
certificate MUST directly certify the one preceding it.”), they obviously occur. Openfire shouldn’t choke on them.
According to the same RFC quote above, the chain should already be ordered. I’m not sure why Openfire explicitly tries to order the certificates from the chain.
If ordering is needed, Openfire could perhaps be modified to order the collection of certificates in a tree-like structure, rather than a list.

guus · March 25, 2016, 10:03am

I have created this item in our issue tracker: [OF-1119] TLS failure when certificate chain is a tree - Jive Software Open Source

guus · March 25, 2016, 11:26am

I’ve created and tested a fix, which will be available in the next release of Openfire (or the next nightly build, if you can’t wait that long).

guus · March 25, 2016, 11:48am

Moments ago, a jabber.org administrator has modified the certificate chain on their end. This will make connectivity possible again with the existing versions of Openfire (assuming that you do add the Let’s Encrypt root certificates to your truststore).

Nicholas_Killewald · March 25, 2016, 3:53pm

Yep, that seemed to do it. Added the root certs for Let’s Encrypt to Openfire, and now it’s talking to jabber.org again. And presumably, once a new release of Openfire hits, the root problem will go away. Thanks!

Michael42 · March 25, 2016, 6:54pm

It is indeed working again since the jabber.org admin modified the certificate chain. Hopefully, the next version of Openfire will be able to parse both linear and tree structured certificate chains correctly. Thank you again Guus for getting involved and creating a test fix so soon!

guus · March 25, 2016, 7:02pm

The next version of Openfire would have worked correctly with the old jabber.org certificate chain. I tested my fix before I informed the jabber.org admins of the problem (after which they fixed the certificate chain on their end). Double whammy!

Michael42 · March 25, 2016, 7:07pm

Security FTW!

LG1 · March 25, 2016, 8:09pm

According to https://letsencrypt.org/certificates/ the certificate is signed by two CAs: “Our intermediate is signed by ISRG Root X1. However, since we are a very new certificate authority, ISRG Root X1 is not yet trusted in most browsers. In order to be broadly trusted right away, our intermediate is also cross-signed by another certificate authority, IdenTrust, whose root is already trusted in all major browsers. Specifically, IdenTrust has cross-signed our intermediate using their DST Root CA X3.”

Openfire should at least log a warn message (better require a property to overrule the RFC) when the CA chain is invalid.