Shaj/PAM/Winbind causes Wildfire crash under heavy load

Dmitriy_Stepanenko · December 23, 2005, 9:33am

To Wildfire developers:

I historically used Jabberd 2.x on FedoraCore 1 as an enterprise IM server in our company. In general it works but seems to be an obscure and difficult-to-control thing. So I looked for alternatives and when I saw Wildfire (it was Jive Messenger that days) learned to use the native authentication (which is vitally necessary in my environment) I’'ve made an attempt to install it and run it instead of Jabberd. It looked cool, an excellent thing! But…

It worked nice until it was connected to the non-standard port and only a few test users were connected to it. (At the same time Jabberd served all the rest of the users on the standard port). The next step was to shutdown Jabberd, start Wildfire on the standard port and let the users (which were suddenly disconnected from the old server) connect to Wildfire.

When I tried to do this, Wildfire could not start. Or, to be more precise, it died about 10 seconds after start. The trace dump it created indicated that the crash occured in “libnss_winbind.so”, which was invoked from “shaj”, which in turn was invoked from Wildfire’‘s user authentication procedures. I’'ve attached an example of this trace dump at the end of this message. Apparently the crash was caused by the situation when about 200 or 300 users simultaneously tried to reconnect and login to the server. So I had to start Jabberd again.

The next morning (today) I’'ve tried to do the same very early, before the most of users even avoke. Wildfire has started without any problem and is running perfectly until now, holding 268 user sessions at this moment.

So, now I can run Wildfire but I cannot restart when many users are connected to the server. (By the way Jabberd can start in this situation although usually it takes about 3-4 minutes to reconnect all user under the full load). Authentication is done through PAM, all other ways does not seem to be acceptable, since the environment is heterogeneous: some users authenticate with ActiveDirectory, others authenticate with OpenLDAP on Linux servers.

What can be done with this?

Java trace dump:

8><----

An unexpected error has been detected by HotSpot Virtual Machine:

SIGSEGV (0xb) at pc=0x4fa4fcd6, pid=13924, tid=2670744

Java VM: Java HotSpot™ Server VM (1.5.0_06-b05 mixed mode)
Problematic frame:
C

T H R E A D -

Current thread (0x53a082f8): JavaThread “Client SR - 5263041” daemon

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x74286f74

Registers:

EAX=0x07ffffff, EBX=0x4fa52100, ECX=0x54286f78, EDX=0x0000001f

ESP=0x54286f38, EBP=0x54287010, ESI=0x54286f78, EDI=0x00000000

EIP=0x4fa4fcd6, CR2=0x74286f74, EFLAGS=0x00010207

Top of Stack: (sp=0x54286f38)

0x54286f38: 000000ee 54286f78 00000000 00000000

0x54286f48: 54286f70 00000000 00000000 00000000

0x54286f58: 00000000 00000000 54286f70 00000005

0x54286f68: 00000000 00000000 00000000 00000000

0x54286f78: 00000000 00000000 00000000 00000000

0x54286f88: 00000000 00000000 00000000 00002000

0x54286f98: 00000000 00000000 00000000 00000000

0x54286fa8: 00000000 00000000 00000000 00000000

Instructions: (pc=0x4fa4fcd6)

0x4fa4fcc6: 8b 93 80 00 00 00 8b 02 89 c2 83 e2 1f c1 e8 05

0x4fa4fcd6: 0f a3 94 85 68 ff ff ff 0f 92 c0 84 c0 74 b5 8b

Stack: [0x5420a000,0x5428a000), sp=0x54286f38, free space=499k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

C

C read_reply+0x39

C winbindd_get_response+0x31

C winbindd_request_response+0x40

C

C write_sock+0x22

C winbindd_send_request+0x81

C winbindd_request_response+0x32

C nsswinbind_getpwnam_r+0x9e

C getpwnam_r+0xd6

C pammodutilgetpwnam+0x67

C unixblankpasswd+0x52

C pam_sm_authenticate+0x116

C

C pamdispatch+0xa0

C

C pam_sm_authenticate+0x2f

C

C pamdispatch+0xa0

C pam_authenticate+0x49

C

C shaj_chkpasswd_pam+0x8c

C Java_com_cenqua_shaj_PAMAuthenticator_checkPasswordImpl+0x87

j com.cenqua.shaj.PAMAuthenticator.checkPasswordImpl(Ljava/lang/String;Ljava/lang /String;Ljava/lang/String;Lcom/cenqua/shaj/log/Log;)Z+0

j com.cenqua.shaj.PAMAuthenticator.checkPAMPassword(Ljava/lang/String;Ljava/lang/ String;Ljava/lang/String;Lcom/cenqua/shaj/log/Log;)Z+55

j com.cenqua.shaj.PAMAuthenticator.checkPassword(Ljava/lang/String;Ljava/lang/Str ing;Ljava/lang/String;Lcom/cenqua/shaj/log/Log;)Z+5

j com.cenqua.shaj.Shaj.checkPassword(Ljava/lang/String;Ljava/lang/String;Ljava/la ng/String;)Z+9

j org.jivesoftware.wildfire.auth.NativeAuthProvider.authenticate(Ljava/lang/Strin g;Ljava/lang/String;)V+6

j org.jivesoftware.wildfire.auth.AuthFactory.authenticate(Ljava/lang/String;Ljava /lang/String;)Lorg/jivesoftware/wildfire/auth/AuthToken;+5

j org.jivesoftware.wildfire.handler.IQAuthHandler.login(Ljava/lang/String;Lorg/do m4j/Element;Lorg/xmpp/packet/IQ;Lorg/xmpp/packet/IQ;Ljava/lang/String;Lorg/jives oftware/wildfire/ClientSession;Ljava/lang/String;)Lorg/xmpp/packet/IQ;+268

j org.jivesoftware.wildfire.handler.IQAuthHandler.handleIQ(Lorg/xmpp/packet/IQ;)L org/xmpp/packet/IQ;+295

j org.jivesoftware.wildfire.handler.IQHandler.process(Lorg/xmpp/packet/Packet;)V+ 7

j org.jivesoftware.wildfire.IQRouter.handle(Lorg/xmpp/packet/IQ;)V+390

j org.jivesoftware.wildfire.IQRouter.route(Lorg/xmpp/packet/IQ;)V+100

j org.jivesoftware.wildfire.PacketRouter.route(Lorg/xmpp/packet/IQ;)V+5

j org.jivesoftware.wildfire.net.SocketReader.processIQ(Lorg/xmpp/packet/IQ;)V+46

j org.jivesoftware.wildfire.net.ClientSocketReader.processIQ(Lorg/xmpp/packet/IQ; )V+13

j org.jivesoftware.wildfire.net.SocketReader.readStream()V+446

j org.jivesoftware.wildfire.net.SocketReader.run()V+72

j java.lang.Thread.run()V+11

v ~StubRoutines::call_stub

V

C

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)