Wildifire 2x

Richard_Kuhn · March 28, 2007, 3:24pm

I recently just moved my Wildfire server to a server machine that runs SCSI Raid 5

It periodically stops allowing connects and needs a reboot. It doesnt show any error messages at all when it does it.

After this happening about 3-4 times I decided to turn on debugging logs to see if I could catch the reason in that.

Well sure enough last night it died out on and I noticed this little block of text.

Any ideas? Or am I looking in the wrong place?

java.net.SocketException: Socket closed

at java.net.SocketOutputStream.socketWrite(Unknown Source)

at java.net.SocketOutputStream.write(Unknown Source)

at java.nio.channels.Channels$WritableByteChannelImpl.write(Unknown Source)

at org.jivesoftware.wildfire.net.TLSStreamWriter.writeToSocket(TLSStreamWriter.jav a:86)

at org.jivesoftware.wildfire.net.TLSStreamWriter.tlsWrite(TLSStreamWriter.java:75)

at org.jivesoftware.wildfire.net.TLSStreamWriter.doWrite(TLSStreamWriter.java:55)

at org.jivesoftware.wildfire.net.TLSStreamWriter.access$100(TLSStreamWriter.java:2 8)

at org.jivesoftware.wildfire.net.TLSStreamWriter$1.write(TLSStreamWriter.java:111)

at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(Unknown Source)

at sun.nio.cs.StreamEncoder$CharsetSE.implWrite(Unknown Source)

at sun.nio.cs.StreamEncoder.write(Unknown Source)

at java.io.OutputStreamWriter.write(Unknown Source)

at java.io.BufferedWriter.flushBuffer(Unknown Source)

at java.io.BufferedWriter.flush(Unknown Source)

at org.jivesoftware.util.XMLWriter.flush(XMLWriter.java:190)

at org.jivesoftware.wildfire.net.XMLSocketWriter.flush(XMLSocketWriter.java:31)

at org.jivesoftware.wildfire.net.SocketConnection.deliver(SocketConnection.java:45 4)

at org.jivesoftware.wildfire.ClientSession.deliver(ClientSession.java:670)

at org.jivesoftware.wildfire.ClientSession.process(ClientSession.java:664)

at org.jivesoftware.wildfire.roster.Roster.broadcastPresence(Roster.java:456)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:243)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:107)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:146)

…edited for space…

org.jivesoftware.wildfire.net.SocketConnection@1e0009b socket: Socket[addr=/10.148.1.113,port=1359,localport=5222] session: org.jivesoftware.wildfire.ClientSession@ec2cf0 status: 3 address: francis.elias@jabber/Exodus id: 7e4c8fd presence:

Message was edited by: rkuhnjr

Richard_Kuhn · March 28, 2007, 3:34pm

After The server stops responding these are the errors i get in the err log

2007.03.27 22:45:43 org.jivesoftware.wildfire.ClientSession.deliver(ClientSession.java:673) Internal server error

org.jivesoftware.wildfire.PacketException: Could not send packet - no route

at org.jivesoftware.wildfire.spi.PacketDelivererImpl.deliver(PacketDelivererImpl.j ava:44)

at org.jivesoftware.wildfire.net.SocketConnection.deliver(SocketConnection.java:46 5)

at org.jivesoftware.wildfire.ClientSession.deliver(ClientSession.java:670)

at org.jivesoftware.wildfire.ClientSession.process(ClientSession.java:664)

at org.jivesoftware.wildfire.roster.Roster.broadcastPresence(Roster.java:456)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:243)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:94)

at org.jivesoftware.wildfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:146)

at org.jivesoftware.wildfire.PresenceRouter.handle(PresenceRouter.java:92)

at org.jivesoftware.wildfire.PresenceRouter.route(PresenceRouter.java:61)

at org.jivesoftware.wildfire.PacketRouter.route(PacketRouter.java:73)

at org.jivesoftware.wildfire.net.SocketReader.processPresence(SocketReader.java:44 4)

at org.jivesoftware.wildfire.net.ClientSocketReader.processPresence(ClientSocketRe ader.java:56)

at org.jivesoftware.wildfire.net.SocketReader.readStream(SocketReader.java:241)

at org.jivesoftware.wildfire.net.SocketReader.run(SocketReader.java:119)

at java.lang.Thread.run(Unknown Source)

LG1 · March 28, 2007, 8:55pm

Hi,

which Wildfire version are you using?

Can you still access the web admin console?

How much free memory does the JVM has? What is the Xmx parameter you are using?

Are your clients using only port 5222 or is port 5223 also open (could be a problem)?

Did you set a parameter to make sure that the JVM uses only IPv4?

Please post also a stack trace of the JVM (kill -3 PID on unix, stacktrace will be written to nohup.out).

LG

Richard_Kuhn · March 29, 2007, 3:26pm

2.4.4

Yes the web admin is still accessible when it dies

Xms = 768 megs

It can be anywhere from 60-80% utilization when it crashes

Clients are only using port 5222 but it is listening on 5223 - How can I stop it?

I have no idea how to set JVM to only use ipv4 so not likley, How would i go about doing this?

if i can catch it when it crashes I will do a stack trace.

It crashed again last night and this is how it is described to me by one of the admins.

1st the rooms go down (you can chat in the room, but not directly message from the room), you can still direct jabber people

2nd you lose the ability to message other people and their status does not update

3rd you lose the ability to authenticate with the server once logged out

Some more background info.

This is a production server that has about 1100 users

the Max concurrent can reach between 500-600 but typically sits around 400

It is a dedicated (older) Server Poweredge 4400

The OS is SLackware 11

Has a gig of ram 768 of which are dedicated to jabber as that is the only real daemon that runs

I also have it setup with its own dedicated Mysql server (So it is jumping the subnet to query the Db it is not local)

It seems to happen about the same time every day when it does happen

At first it was right around 3:30 pm now it seems to have moved to 9 at night or so.

LG1 · March 29, 2007, 8:36pm

Hi,

Security Settings - http://server:9091/ssl-settings.jsp allows you to select “Custom” and there “Old SSL method: Not Available”, this will disable port 5223. But this issue exists as far as I know only with Wildfire 3.2.x.

Set “-Djava.net.preferIPv4Stack=true” just as you set “-Xmx768m”.

Do you have only 1 GB physical memory? If the JVM uses 768 MB for the java heap then you have 256 MB for the operating system and for the native java heap, for me this seems to be not enough. The native heap must manage 550-650 threads and it may also fragment.

Is it an option for you to add more physical memory to the server or to migrate to Wildfire 3.2.3 which uses Java NIO for less threads?

It may also help to set Xmx to 700 MB, but there’'s a higher risk to get out of memory errors.

LG

Richard_Kuhn · March 30, 2007, 1:47pm

Actually I have reduced it as much as 512m

I got no out of error messages but it still crashed last night

The admin was able to capture these logs

(I wasnt there to do a stack trace)

2007.03.29 19:30:09 org.jivesoftware.wildfire.net.SocketReader.negotiateTLS(SocketReader.java:681) Error while negotiating TLS

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(Unknown Source)

at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(Unknown Source)

at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(Unknown Source)

at javax.net.ssl.SSLEngine.unwrap(Unknown Source)

at org.jivesoftware.wildfire.net.TLSStreamHandler.doHandshake(TLSStreamHandler.jav a:207)

at org.jivesoftware.wildfire.net.TLSStreamHandler.(TLSStreamHandler.java:145)

at org.jivesoftware.wildfire.net.SocketConnection.startTLS(SocketConnection.java:1 50)

at org.jivesoftware.wildfire.net.SocketReader.negotiateTLS(SocketReader.java:678)

at org.jivesoftware.wildfire.net.SocketReader.readStream(SocketReader.java:267)

at org.jivesoftware.wildfire.net.SocketReader.run(SocketReader.java:119)

at java.lang.Thread.run(Unknown Source)

2007.03.29 19:30:09 org.jivesoftware.wildfire.net.SocketReader.run(SocketReader.java:159) Connection closed before session established

Socket[addr=/10.148.1.129,port=3963,localport=5222]

2007.03.29 21:00:30 org.jivesoftware.wildfire.net.SocketReader.negotiateTLS(SocketReader.java:681) Error while negotiating TLS

javax.net.ssl.SSLException: Inbound closed before receiving peer’'s close_notify: possible truncation attack?

at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)

at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(Unknown Source)

at com.sun.net.ssl.internal.ssl.SSLEngineImpl.closeInbound(Unknown Source)

at org.jivesoftware.wildfire.net.TLSStreamHandler.doHandshake(TLSStreamHandler.jav a:198)

at org.jivesoftware.wildfire.net.TLSStreamHandler.(TLSStreamHandler.java:145)

at org.jivesoftware.wildfire.net.SocketConnection.startTLS(SocketConnection.java:1 50)

at org.jivesoftware.wildfire.net.SocketReader.negotiateTLS(SocketReader.java:678)

at org.jivesoftware.wildfire.net.SocketReader.readStream(SocketReader.java:267)

at org.jivesoftware.wildfire.net.SocketReader.run(SocketReader.java:119)

at java.lang.Thread.run(Unknown Source)

2007.03.29 21:00:30 org.jivesoftware.wildfire.net.SocketReader.run(SocketReader.java:159) Connection closed before session established

Socket[addr=/10.148.1.185,port=1394,localport=5222]

Last night it crashed with Xmx = 512

and memory was at 42%

Eventually I will likley be upgrading everyone to spark and the server to a 3x version if this keeps, was hoping to make till a “Real” New server showed up.

Message was edited by: rkuhnjr