SSL auth stopped working

I received an email from a user stating that openfire was not working correctly.

When I attempted to login received invalid u/p error.

I saw in logs the following:

2015.11.04 20:08:08 org.jivesoftware.openfire.ldap.LdapGroupProvider - simple bind failed: <SNIP_SERVER>.:636

javax.naming.CommunicationException: simple bind failed: <SNIP_SERVER>.:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

at com.sun.jndi.ldap.LdapClient.authenticate(Unknown Source)

at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)

at com.sun.jndi.ldap.LdapCtx.(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)

at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)

at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)

at javax.naming.InitialContext.init(Unknown Source)

at javax.naming.ldap.InitialLdapContext.(Unknown Source)

at org.jivesoftware.util.JiveInitialLdapContext.(JiveInitialLdapContext.java :43)

at org.jivesoftware.openfire.ldap.LdapManager.getContext(LdapManager.java:548)

at org.jivesoftware.openfire.ldap.LdapManager.findGroupDN(LdapManager.java:1101)

at org.jivesoftware.openfire.ldap.LdapManager.findGroupDN(LdapManager.java:1055)

at org.jivesoftware.openfire.ldap.LdapGroupProvider.getGroup(LdapGroupProvider.jav a:82)

at org.jivesoftware.openfire.group.GroupManager.getGroup(GroupManager.java:343)

at org.jivesoftware.openfire.group.GroupManager.getGroup(GroupManager.java:320)

at org.jivesoftware.openfire.group.GroupCollection$GroupIterator.getNextElement(Gr oupCollection.java:113)

at org.jivesoftware.openfire.group.GroupCollection$GroupIterator.hasNext(GroupColl ection.java:76)

at org.jivesoftware.openfire.roster.RosterManager.getSharedGroups(RosterManager.ja va:191)

at org.jivesoftware.openfire.roster.Roster.(Roster.java:128)

at org.jivesoftware.openfire.roster.RosterManager.getRoster(RosterManager.java:116 )

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:307)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:162)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:137)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:201)

at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:148)

at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84)

at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1242)

at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:292)

at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:275)

at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:224)

at org.jivesoftware.openfire.nio.NIOConnection.systemShutdown(NIOConnection.java:2 82)

at org.jivesoftware.openfire.spi.LocalRoutingTable.stop(LocalRoutingTable.java:146 )

at org.jivesoftware.openfire.spi.RoutingTableImpl.stop(RoutingTableImpl.java:953)

at org.jivesoftware.openfire.XMPPServer.shutdownServer(XMPPServer.java:995)

at org.jivesoftware.openfire.XMPPServer.access$800(XMPPServer.java:148)

at org.jivesoftware.openfire.XMPPServer$ShutdownHookThread.run(XMPPServer.java:941 )

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at sun.security.ssl.Alerts.getSSLException(Unknown Source)

at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)

at sun.security.ssl.Handshaker.fatalSE(Unknown Source)

at sun.security.ssl.Handshaker.fatalSE(Unknown Source)

at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)

at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)

at sun.security.ssl.Handshaker.processLoop(Unknown Source)

at sun.security.ssl.Handshaker.process_record(Unknown Source)

at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)

at sun.security.ssl.AppOutputStream.write(Unknown Source)

at java.io.BufferedOutputStream.flushBuffer(Unknown Source)

at java.io.BufferedOutputStream.flush(Unknown Source)

at com.sun.jndi.ldap.Connection.writeRequest(Unknown Source)

at com.sun.jndi.ldap.Connection.writeRequest(Unknown Source)

at com.sun.jndi.ldap.LdapClient.ldapBind(Unknown Source)

… 40 more

Nothing changed in the software (all admins went home, no one was logged into the server) OS updates hadn’t happened yet for that day.

I looked at the keystore file and the cert file for the server listed above was still there.

./keytool -list -alias <SNIPPED_SERVER> -keystore …/lib/security/cacerts

Enter keystore password:

<SNIPPED_SERVER>, Jul 1, 2015, trustedCertEntry,

Certificate fingerprint (SHA1): 81:BC:90:59:41:CD:F4:C8:8B:6B:D9:FA:BB:F4:76:81:76:3E:D9:68

This cert does not expire until December.

Prior to this we see this in logs.

2015.11.04 17:16:47 org.jivesoftware.openfire.nio.NIOConnection - Failed to deliver packet: IT2nd Floor

2015.11.04 17:16:47 org.jivesoftware.openfire.session.LocalSession - Internal server error

java.lang.IllegalStateException: Connection closed

java.lang.IllegalStateException: Connection closed

at org.jivesoftware.openfire.nio.NIOConnection.deliver(NIOConnection.java:316)

at org.jivesoftware.openfire.session.LocalClientSession.deliver(LocalClientSession .java:857)

at org.jivesoftware.openfire.session.LocalSession.process(LocalSession.java:289)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToLocalDomain(RoutingTableI mpl.java:354)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:239)

at org.jivesoftware.openfire.SessionManager.userBroadcast(SessionManager.java:1068 )

at org.jivesoftware.openfire.roster.Roster.broadcast(Roster.java:685)

at org.jivesoftware.openfire.roster.Roster.broadcast(Roster.java:718)

at org.jivesoftware.openfire.roster.Roster.(Roster.java:163)

at org.jivesoftware.openfire.roster.RosterManager.getRoster(RosterManager.java:116 )

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:307)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:162)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:137)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:201)

at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:148)

at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84)

at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1242)

at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:292)

at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:275)

at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:224)

at org.jivesoftware.openfire.nio.NIOConnection.deliverRawText(NIOConnection.java:3 96)

at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:246)

at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:224)

at org.jivesoftware.openfire.nio.NIOConnection.deliverRawText(NIOConnection.java:3 96)

at org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHandl er.java:154)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.exceptionCaugh t(DefaultIoFilterChain.java:672)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextExceptionCaught(D efaultIoFilterChain.java:461)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1100(DefaultIoFilt erChain.java:47)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.exceptionCaug ht(DefaultIoFilterChain.java:760)

at org.apache.mina.core.filterchain.IoFilterAdapter.exceptionCaught(IoFilterAdapte r.java:102)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextExceptionCaught(D efaultIoFilterChain.java:461)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1100(DefaultIoFilt erChain.java:47)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.exceptionCaug ht(DefaultIoFilterChain.java:760)

at org.apache.mina.core.filterchain.IoFilterAdapter.exceptionCaught(IoFilterAdapte r.java:102)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextExceptionCaught(D efaultIoFilterChain.java:461)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1100(DefaultIoFilt erChain.java:47)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.exceptionCaug ht(DefaultIoFilterChain.java:760)

at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:93)

at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(Ordere dThreadPoolExecutor.java:769)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(Order edThreadPoolExecutor.java:761)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThr eadPoolExecutor.java:703)

at java.lang.Thread.run(Unknown Source)

Server is RHEL6, running openfire 3.10.2

I do not have an admin login as everything is via LDAP. I have attempted to use what is set in Resetting admin passwords however we don’t set a password there and I cannot get access to a location to place a password. My password doesn’t work since I auth via ldap.

I deleted and readded the cert, this didn’t seem to do anything, still error messages.

2015.11.04 22:03:11 org.jivesoftware.openfire.ldap.LdapAuthProvider - Error connecting to LDAP server

javax.naming.CommunicationException: simple bind failed: .:636 [Root exception is javax.net.ssl.SSLHandshakeException:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find

valid certification path to requested target]

at com.sun.jndi.ldap.LdapClient.authenticate(Unknown Source)

at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)

at com.sun.jndi.ldap.LdapCtx.(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)

at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)

at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)

at javax.naming.InitialContext.init(Unknown Source)

at javax.naming.ldap.InitialLdapContext.(Unknown Source)

at org.jivesoftware.util.JiveInitialLdapContext.(JiveInitialLdapContext.java :43)

at org.jivesoftware.openfire.ldap.LdapManager.getContext(LdapManager.java:548)

at org.jivesoftware.openfire.ldap.LdapManager.findUserDN(LdapManager.java:955)

at org.jivesoftware.openfire.ldap.LdapManager.findUserDN(LdapManager.java:908)

at org.jivesoftware.openfire.ldap.LdapAuthProvider.authenticate(LdapAuthProvider.j ava:123)

at org.jivesoftware.openfire.auth.AuthFactory.authenticate(AuthFactory.java:213)

at org.jivesoftware.openfire.net.XMPPCallbackHandler.handle(XMPPCallbackHandler.ja va:102)

at org.jivesoftware.openfire.sasl.SaslServerPlainImpl.evaluateResponse(SaslServerP lainImpl.java:120)

at org.jivesoftware.openfire.net.SASLAuthentication.handle(SASLAuthentication.java :306)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:173)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:177)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceive d(DefaultIoFilterChain.java:690)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(D efaultIoFilterChain.java:417)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilt erChain.java:47)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceiv ed(DefaultIoFilterChain.java:765)

at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapte r.java:109)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(D efaultIoFilterChain.java:417)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilt erChain.java:47)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceiv ed(DefaultIoFilterChain.java:765)

at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flus h(ProtocolCodecFilter.java:407)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:236)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(D efaultIoFilterChain.java:417)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilt erChain.java:47)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceiv ed(DefaultIoFilterChain.java:765)

at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:74)

at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(Ordere dThreadPoolExecutor.java:769)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(Order edThreadPoolExecutor.java:761)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThr eadPoolExecutor.java:703)

at java.lang.Thread.run(Unknown Source)

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at sun.security.ssl.Alerts.getSSLException(Unknown Source)

at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)

at sun.security.ssl.Handshaker.fatalSE(Unknown Source)

at sun.security.ssl.Handshaker.fatalSE(Unknown Source)

at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)

at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)

at sun.security.ssl.Handshaker.processLoop(Unknown Source)

at sun.security.ssl.Handshaker.process_record(Unknown Source)

at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)

at sun.security.ssl.AppOutputStream.write(Unknown Source)

at java.io.BufferedOutputStream.flushBuffer(Unknown Source)

at java.io.BufferedOutputStream.flush(Unknown Source)

at com.sun.jndi.ldap.Connection.writeRequest(Unknown Source)

at com.sun.jndi.ldap.Connection.writeRequest(Unknown Source)

at com.sun.jndi.ldap.LdapClient.ldapBind(Unknown Source)

… 41 more

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at sun.security.validator.PKIXValidator.doBuild(Unknown Source)

at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)

at sun.security.validator.Validator.validate(Unknown Source)

at sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)

… 54 more

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)

at java.security.cert.CertPathBuilder.build(Unknown Source)

… 60 more

Rob,

Make sure you are loading the root cert into the correct keystore.

3.10.2 with LDAPS (LDAP over SSL)

ok that did it. Any clue why it would randomly just stop working? We imported the original cert in July and it’s been running (and restarted) a bunch of times since then. The SSL cert imported doesn’t expire until the middle of december?

my guess would be either the keystore become corrupt, or if OF is using a systemwide java and not the included jre, perhaps java was removed and reinstalled

alright not sure, it’s using the local java, and that wasn’t touched (nor was openfire)

i could read from the keystore, removed and readded it, but it never needed our internal CA cert until now. Good to know for the future, just odd it randomly showed up after folks were gone and saw the “internal errors” in the logs.

Thanks for the help, it is appreciated.