Openfire LDAP logons fail randomly

macado · March 25, 2013, 8:22pm

We used to run Openfire 3.7.1 on a Windows VM but recently moved our Openfire server to a Linux VM running Openfire 3.8.1. We’ve been having random login issues. We had no issues on the previous version, the only changes are that it’s now running 3.8.1 and under linux as opposed to winodws.

-bash-4.1$ uname -a

Linux servername 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

Java Version:
1.6.0_41 Sun Microsystems Inc. – Java HotSpot™ Server VM
Appserver:
jetty/7.x.y-SNAPSHOT
Host Name:
servername
OS / Hardware:
Linux / i386
Locale / Timezone:
en / Eastern Standard Time (-5 GMT)
Java Memory

46.60 MB of 454.38 MB (10.3%) used

It is binded to Active Directory and uses a AD security group (with nested security groups) for authenication. It seems like the LDAP login times out. We are binding to port 389 since it seems like LDAP over SSL 636 was even slower.

Users/Group takes forever to display correctly and will sometimes time out and just be completely blank. There are approximately 501 users.

ldap.searchFilter (&(objectclass=organizationalPerson)(|(memberOf:1.2.840.113556.1.4.1941:=CN=UIT -Jabber-Users,**removed,DC=edu)))

Info Logs will display tons of User Login Failed. PLAIN authenicated failed for: user

2013.03.25 09:11:53 org.jivesoftware.openfire.net.SASLAuthentication - User Login Failed. PLAIN authentication failed for: user

2013.03.25 09:13:20 org.jivesoftware.openfire.net.SASLAuthentication - User Login Failed. PLAIN authentication failed for: user

2013.03.25 09:13:46 org.jivesoftware.openfire.net.SASLAuthentication - User Login Failed. PLAIN authentication failed for: user

2013.03.25 09:14:08 org.jivesoftware.openfire.net.SASLAuthentication - User Login Failed. PLAIN authentication failed for: user

javax.naming.CommunicationException: ***name removed.edu:389 [Root exception is java.net.ConnectException: Connection timed out]

at com.sun.jndi.ldap.Connection.(Unknown Source)

at com.sun.jndi.ldap.LdapClient.(Unknown Source)

at com.sun.jndi.ldap.LdapClient.getInstance(Unknown Source)

at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)

at com.sun.jndi.ldap.LdapCtx.(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)

at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)

at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)

at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)

at javax.naming.InitialContext.init(Unknown Source)

at javax.naming.ldap.InitialLdapContext.(Unknown Source)

at org.jivesoftware.util.JiveInitialLdapContext.(JiveInitialLdapContext.java :43)

at org.jivesoftware.openfire.ldap.LdapManager.getContext(LdapManager.java:535)

at org.jivesoftware.openfire.ldap.LdapManager.retrieveList(LdapManager.java:1838)

at org.jivesoftware.openfire.ldap.LdapUserProvider.getUsernames(LdapUserProvider.j ava:177)

at org.jivesoftware.openfire.user.UserManager.getUsernames(UserManager.java:266)

at org.jivesoftware.openfire.roster.RosterManager.getSharedUsersForRoster(RosterMa nager.java:849)

at org.jivesoftware.openfire.roster.Roster.getSharedUsers(Roster.java:658)

at org.jivesoftware.openfire.roster.Roster.(Roster.java:148)

at org.jivesoftware.openfire.roster.RosterManager.getRoster(RosterManager.java:116 )

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:305)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:147)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:135)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:199)

at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:148)

at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84)

at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java: 355)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:100)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:272)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:194)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:181)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

Caused by: java.net.ConnectException: Connection timed out

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(Unknown Source)

at java.net.PlainSocketImpl.connectToAddress(Unknown Source)

at java.net.PlainSocketImpl.connect(Unknown Source)

at java.net.SocksSocketImpl.connect(Unknown Source)

at java.net.Socket.connect(Unknown Source)

at java.net.Socket.(Unknown Source)

at com.sun.jndi.ldap.Connection.createSocket(Unknown Source)

… 53 more

Any ideas?

M_Tejera · March 26, 2013, 9:41am

Linux + Openfire + Active Directory has some weird issues as far as i know.

I would recommend to change to Windows if you’re planning to use AD for authentication.

macado · March 26, 2013, 8:27pm

Thanks. For some reason I switched the AD/LDAP binding port to 3268 and it seems to have helped tremendously. This has made User/Group search much faster and I dont seem to be getting as many user login failed errors.

Any idea why this would be the case?

macado · March 27, 2013, 3:55pm

It definitely helped but did not fix the problem. Any other tweaks/configuration changes I can try?

David29 · March 27, 2013, 5:31pm

Do you have ldap.connectionPoolEnabled set to true in system properties? Are you pointing Openfire to a specific domain controller, or to a hostname which points to multiple (e.g. domain.local, that contains all of your DCs?).

LDAP connection pooling doesn’t work with SSL, so it’s best to leave that off - I personally use stunnel to build a SSL connection to our AD environment, then have Openfire connect to the local end of the tunnel. Gives me SSL on the wire without OF having to support it properly.

You may also want to check with tcpdump to see if your connection timeouts are related to the initial connection to the DC, or when it runs a query. May indicate a network issue somewhere.

We run OF on Linux (RHEL6) with a Win2008 AD environment and have not experienced any issues, so it is a workable solution.

speedy · March 27, 2013, 5:37pm

I would try upgrading to java 1.7. java 1.6 on linux seems to be a little buggy with AD authentication

David29 · March 27, 2013, 7:53pm

We still run on Java 1.6 - Tried both Oracle and OpenJDK. Both work fine.

What problems did you have with AD authentication with Java 1.6?

speedy · March 28, 2013, 1:19pm

mostly with SSO, but while researching that, I came across alot of other post of users with weird AD/java 1.6 related issue that upgrading to 1.7 (most upgraded to openjdk 1.7) resolved.

macado · March 28, 2013, 5:40pm

I will try upgrading to Java 7. Anyone else had the same experience? You said that you have less LDAP/AD issues with Java 7?

speedy · March 28, 2013, 11:36pm

I just noticed you are running 3.8.1 to where you were running 3.7.1 Another issue COULD be a bug with 3.8.1. If updating java doesn’t help, you may try downgrading to openfire 3.7.1 to see if that resolves your issue.

macado · April 9, 2013, 5:12pm

Had the same issue with Openfire 3.7.1 on new server.

I updated to Java 7 and the issue is still occuring. The issue seems to happen less than before but I still get PLAIN authenication failed. It will eventually let the user log in.

2013.04.09 13:10:01 org.jivesoftware.openfire.net.SASLAuthentication - User Login Failed. PLAIN authentication failed for: username

macado · April 9, 2013, 5:28pm

I am pointing to domain fqdn which points to all DCs and I have ldap.connectionPoolEnabled set to true.

Have tried with LDAP 389 and over LDAPS 636 also.

speedy · April 9, 2013, 5:32pm

try pointing to a single DC. Also, im curious…is it the same user accounts or different ones?

macado · April 9, 2013, 5:58pm

All different user accounts. Not one in particular. (Completely random). They’re all in varying OUs but all part of the search filter group.

speedy · April 9, 2013, 8:08pm

you might also put wireshark on a workstation thats having the problems to see if you can narrow down the issue.

macado · April 12, 2013, 1:54pm

I think I finally have this working stable so it’s not rejecting logins. Rather than binding to domain.ad which resolves to what domain controller it hits, If I bind to one specific AD then it works. Any idea why this would be the case?

This problem only seems to occur when I run Openfire on a Linux VM. It didnt cause any issues when Openwire was running on Windows.

I’m not sure if that was the actual cause but in any case, binding a specific domain controller seems to have solved the issue for us.

Changes made:

Upgraded to JRE 7

Bind to LDAP/AD port 3268 instead of 326

Bind to a specific domain controller rather than domain.local