Admin console not working -- listening, but not responding

Manuzhai · June 17, 2006, 5:41pm

Okay, I hope you guys can help me out with this.

Earlier today, I tried installing Wildfire 2.6.2 on my (Gentoo) Linux server. I installed SUN JDK 1.5.0.06, downloaded the tarball and unzipped it to /opt/wildfire. I then started the server and used an SSH tunnel to get into the admin console. I got most stuff working, except SSL. I wanted to start over with the default SSL certificates, and figured out I should start over with the entire thing (I’'m using the MySQL provider, BTW, with MySQL 4.1.20 running on the same box).

So: I rm-rf’‘d /opt/wildfire and unzipped the tarball to it again. I started the server and established the SSH tunnel again. However, it doesn’‘t seem to respond. The info.log says it’‘s listening, there is nothing much in the error logs, and I can see it listening using netstat -a, but when I request the page either through an SSH tunnel or using elinks on the server or through mod_proxy on Apache, it just doesn’'t respond.

It also seems to take three (3!) minutes to go from starting to listening. Is this normal? Why was it all working earlier this morning, and why won’'t it respond now?

Any tips would be very much appreciated.

LG1 · June 17, 2006, 9:02pm

Hi,

did you stop Wildfire before rm -rf? I wonder if you still have some zombie processes.

It also seems to take three (3!) minutes to go from starting to listening. Is this normal? /i

Do you have a process which is looping or a 200 MHz server?

I assume you are not using root to unzip and run wildfire, so you could check the directory permissions just to make sure that these are right.

In this early stage no db connection should have been established or configured so I wonder what is going on. Could you please check if top shows a process with 100% cpu, and if it is Wildfire/Java use a “kill -3 PID” to obtain a thread dump (should be saved in nohup.out).

LG

Manuzhai · June 17, 2006, 9:24pm

did you stop Wildfire before rm -rf? I wonder if you still have some zombie processes.

I did monitor any Java processes (don’'t have any other Java apps) at all times using ps -Af.

Do you have a process which is looping or a 200 MHz server?

Looping process? No idea, how could I check that. It’‘s a 3Ghz Xeon (512MB, enough disk space), so that shouldn’'t be the problem.

I assume you are not using root to unzip and run wildfire, so you could check the

directory permissions just to make sure that these are right.

I am[/i] using root to do all this for now. What directories should I check?

Stopping and starting using “wildfire start” and “wildfire stop” seems to work quite well, the process appears/disappears promptly. It has a short CPU spike at startup, then goes down to 0% CPU. It uses about 9% of memory (so that would be about 5MB).

I’'ve put my logs and two thread-dumps at http://manuzhai.nl/wildfire/. before-listening.log is a thread dump from before it is even listening to *:9090, while-listening.log is from while it is listening.

I hope this helps!

LG1 · June 18, 2006, 11:09am

Hi,

I wonder how “wildfire” was added to your local path, I always use the absolute path as the wildfireHome environment may otherwise be not found.

Using root to install and start it is really not necessary but there should be no issues with file permissions in this case.

Run top or vmstat to check if the server is idle or if a process is looping, but this does not seem to be the case

So I have no idea what is failing, maybe some debug log statements help to identify a problem. Add

in conf/wildfire.xml to get them.

LG

Manuzhai · June 18, 2006, 11:29am

Hmm, it looks like this would be the problem:

13:29:39.452 TRACE [Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=9090]] org.mortbay.util.LogSupport.ignore(LogSupport.java:36) >03> IGNORED

java.net.SocketTimeoutException: Accept timed out

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

at java.net.ServerSocket.implAccept(ServerSocket.java:450)

at java.net.ServerSocket.accept(ServerSocket.java:421)

at org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)

at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:634)

13:29:40.160 TRACE [Acceptor [SSL: ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=9091]]] org.mortbay.util.LogSupport.ignore(LogSupport.java:36) >03> IGNORED

java.net.SocketTimeoutException: Accept timed out

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

at java.net.ServerSocket.implAccept(ServerSocket.java:450)

at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.jav a:259)

at org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)

at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:634)

(There are lots and lots of these…) But why does it do that?

Manuzhai · June 18, 2006, 11:37am

Here’‘s the part that’'s taking a LOT of time:

13:25:50.271 INFO org.mortbay.util.Container.start(Container.java:74) >16> Started WebApplicationContext[/,Wildfire]

13:28:59.408 INFO org.mortbay.http.SocketListener.start(SocketListener.java:204) >16> Started SocketListener on 0.0.0.0:9090

BTW, I was just starting it from inside /opt/wildfire/bin, using “wildfire start”.

LG1 · June 18, 2006, 2:31pm

Hi,

for me it takes some ms during startup:

16:00:55.595 INFO org.mortbay.util.Container.start(Container.java:74) >16> Started WebApplicationContext[/

,Wildfire]

16:00:55.599 INFO org.mortbay.http.SocketListener.start(SocketListener.java:204) >16> Started SocketListen

er on 0.0.0.0:9090

So I have currently no idea what did change that it takes so long for you to start.

LG

Manuzhai · June 18, 2006, 2:36pm

Too bad. I hope one of the developers has an idea…

Manuzhai · June 19, 2006, 11:40am

Looks like this guy has the same problem: http://www.jivesoftware.org/community/message.jspa?messageID=115868

LG1 · June 19, 2006, 1:56pm

Hi,

these TRACE messages are normal. I see them every 10 seconds for port 9090 and 9091.

LG

Manuzhai · June 19, 2006, 2:07pm

Oh; so it just listens for ten seconds, then times out, and listens again?

Okay, but then what’‘s the problem? This stuff is driving me crazy. I think it must be something else I changed on my box, but I’'ve no clue what that could be.

Is there anything wildfire touches outside of the /opt/wildfire directory? The database shouldn’‘t matter, since it hasn’'t been configured yet at the time of the problem.

And thanks for all the help, really, it’‘s just frustrating that I don’‘t understand why it doesn’'t work.

Manuzhai · June 21, 2006, 1:55pm

Thanks to Guus, who was idling in dev@conference.jivesoftware.com, I’'ve been able to solve this.

Mostly, it turns out that this was a problem with a firewall which was a little too restrictive. However, starting the server still takes more than 3 minutes for me! See this console transcript:

enrai logs # date && …/bin/wildfire start

Wed Jun 21 15:46:05 CEST 2006

Starting wildfire

enrai logs # tail -f info.log

2006.06.21 15:45:40 Server halted

2006.06.21 15:49:16 Started server (unencrypted) socket on port: 5269

2006.06.21 15:49:16 Started plain (unencrypted) socket on port: 5222

2006.06.21 15:49:17 Started SSL (encrypted) socket on port: 5223

2006.06.21 15:49:17 Publish-Subscribe domain: pubsub.manuzhai.nl

2006.06.21 15:49:17 Multi User Chat domain: conference.manuzhai.nl

2006.06.21 15:49:17 Wildfire 2.6.2

2006.06.21 15:49:20 Admin console listening at:

http://manuzhai.nl:9090

https://manuzhai.nl:9091

(I made sure the old server wasn’‘t running and it wasn’‘t listening on any ports before starting the new one.) This is a tad long, right? And it’‘s weird how it doesn’'t log ANYTHING (even with the log.debug.enabled = true) for the first 3 minutes.

Also, it’'s consistenly throwing this NullServerException at startup:

Internal server error

java.lang.NullPointerException

at org.jivesoftware.wildfire.disco.IQDiscoItemsHandler.addServerItemsProvider(IQDi scoItemsHandler.java:178)

at org.jivesoftware.wildfire.disco.IQDiscoItemsHandler.start(IQDiscoItemsHandler.j ava:270)

at org.jivesoftware.wildfire.XMPPServer.startModules(XMPPServer.java:497)

at org.jivesoftware.wildfire.XMPPServer.start(XMPPServer.java:371)

at org.jivesoftware.wildfire.XMPPServer.(XMPPServer.java:142)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessor Impl.java:39)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor AccessorImpl.java:27)

at java.lang.reflect.Constructor.newInstance(Constructor.java:494)

at java.lang.Class.newInstance0(Class.java:350)

at java.lang.Class.newInstance(Class.java:303)

at org.jivesoftware.wildfire.starter.ServerStarter.start(ServerStarter.java:88)

at org.jivesoftware.wildfire.starter.ServerStarter.main(ServerStarter.java:49)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja va:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)

at com.install4j.runtime.Launcher.main(Unknown Source)