Webchat running in Tomcat - connection was cut?

Openfire Openfire Support
1

It seems that every now and again Tomcat disallows new connections (clicks on our “available to chat” icon). I was wondering if anyone might know what was going on here.

This is from the Tomcat webchat-error.log…

Feb 16, 2007 9:21:29 AM com.jivesoftware.webchat.util.WebLog logError

WARNING: Failed writing out image on Fri Feb 16 09:21:29 PST 2007

ClientAbortException: java.net.SocketException: Connection reset

at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:327)

at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:293)

at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java: 97)

at com.jivesoftware.webchat.util.SettingsManager.writeBytesToStream(SettingsManage r.java:90)

at com.jivesoftware.webchat.LiveAssistantServlet.service(LiveAssistantServlet.java :177)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFil terChain.java:252)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain .java:173)

at com.jivesoftware.webchat.SetCharacterEncodingFilter.doFilter(SetCharacterEncodi ngFilter.java:44)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFil terChain.java:202)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain .java:173)

at com.jivesoftware.webchat.SetupFilter.doFilter(SetupFilter.java:91)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFil terChain.java:202)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain .java:173)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java: 213)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java: 178)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:10 7)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)

at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConn ection(Http11BaseProtocol.java:664)

at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:5 27)

at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorke rThread.java:80)

at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:6 84)

at java.lang.Thread.run(Thread.java:595)

Caused by: java.net.SocketException: Connection reset

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)

at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:283)

at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:272)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:663)

at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)

at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuff er.java:746)

at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)

at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:3 04)

at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:991)

at org.apache.coyote.Response.action(Response.java:182)

at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:322)

… 25 more

2

Is this caused because Tomcat has too high of a load? I’'d love to get this resolved, and, if I need to, I can throw another server at this…

3

I do not know if these errors in the Wildfire error log are related, however here they are (since they are repeating…)

2007.02.16 09:56:18 org.jivesoftware.wildfire.session.OutgoingServerSession.createOutgoingSession(Ou tgoingServerSession.java:258) Error trying to connect to remote server: null(DNS lookup: null:5269)

java.net.UnknownHostException: null

at java.net.PlainSocketImpl.connect(Unknown Source)

at java.net.SocksSocketImpl.connect(Unknown Source)

at java.net.Socket.connect(Unknown Source)

at org.jivesoftware.wildfire.session.OutgoingServerSession.createOutgoingSession(O utgoingServerSession.java:253)

at org.jivesoftware.wildfire.session.OutgoingServerSession.authenticateDomain(Outg oingServerSession.java:142)

at org.jivesoftware.wildfire.server.OutgoingSessionPromise$PacketsProcessor.sendPa cket(OutgoingSessionPromise.java:199)

at org.jivesoftware.wildfire.server.OutgoingSessionPromise$PacketsProcessor.run(Ou tgoingSessionPromise.java:184)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

4

Hey Matthew,

Are you running Tomcat and Wildfire in the same machine? Or is there a firewall between them? Sometimes firewalls are configured to close idle connections or connections that have “lived for too long”. BTW, just to discard the high load possibility it would be nice if you can run the webchat client in another machine.

Regards,

– Gato

5

Yes, tomcat / wildfire / mysql are all on the same machine. Here are some stats on the hardware:

load average: 0.14, 0.13, 0.39

Mem: 2074652k total, 1782496k used, 292156k free

9946 root 16 0 989m 563m 15m S 11.7 27.8 852:23.91 java (this is wildfire, 11.7% CPU, 27.8% mem)

31995 root 25 0 852m 248m 8160 S 1.7 12.3 54:01.79 java (this is tomcat, 1.7% CPU, 12.3% mem)

11165 mysql 16 0 948m 142m 4876 S 1.0 7.0 3575:06 mysqld-max

Here are the JVM settings for Tomcat (wildfire is all “stock”)

Free memory: 48.62 MB Total memory: 466.50 MB Max memory: 616.31 MB

And the information for Tomcat port 443 (we run everything over ssl)

Max threads: 300 Min spare threads: 25 Max spare threads: 75 Current thread count: 219 Current thread busy: 142

I did connect via chat.zappos.com when I set up the webapp in Tomcat. This address resolves to our firewall, so even though the software is all on the same machine traffic still goes out and through the firewall and back into the server.

Do you think this problem is intermittent and would fix itself? Am I just having “bad luck” with getting the errors? The weirdest part is that jivelive.jsp loads and runs fine - the chat icon shows as it should - but then when clicked on it does not load (gives the no connection message after a few seconds).

6

When you say it gives the “no connection message after a few seconds”, is it no connection to wildfire or no http connection?

Thanks,

Derek

7

Hey Matthew,

Which version of Wildfire, Wildfire Enterprise and webchat.war are you using?

Thanks,

– Gato

8

No connection to wildfire. I actually do get a nice printout of a Java error where the information form would be.

This is all 3.2.0 version products. I am going to upgrade to 3.2.1 tomorrow.

It seems that the webchat.war is having trouble connecting to Wildfire when under heavy load. By heavy load, for example, we had 17 million GET requests from 2am to 3am yesterday. While a lot of those would be images, some are pages - and of those pages, around 1/4 are chat enabled.

9

(I’‘m getting the feeling that we’'re your largest click to chat deployment yet… heh)

At any rate. I’'m wondering if perhaps adding a connection manager to the server may help. While we only have a handful of agents logged in (8 or 9), we have millions of requests coming in for jivelive.jsp. I assume that this means that we have lots and lots of webchat.war -> wildfire XMMP connections open.

In tomcat, today, I noticed that there are around 850 active webchat app sessions - if we translate that to chat users then we’'d normally have a connection manager right? I wish we could get Wildfire running smoothly!

A perl version of jivelive.jsp would be FANTASTIC so that we could deploy it on our normal web servers.

That said, I’‘m also looking into a way to cache jivelive.jsp, so that it’‘s not so intensive to load it. I don’'t want that load to have to talk to the wildfire server if at all possible (except for every few minutes to check on the status of chat agents).

10

As this just happened a moment ago…

“No response from server on status set” is the actual error I get. Reloading webchat inside Tomcat fixxes the issue.

11

This is the actual error message that we get…

Online Chat Service

Our chat service is unavailable at this time. Please check back soon.

No response from server on status set.: at com.jivesoftware.smack.workgroup.user.Workgroup.getWorkgroupForm(Unknown Source) at org.apache.jsp.userinfo_jsp._jspService(userinfo_jsp.java:91) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFil terChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain .java:173) at com.jivesoftware.webchat.SetCharacterEncodingFilter.doFilter(SetCharacterEncodi ngFilter.java:44) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFil terChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain .java:173) at com.jivesoftware.webchat.SetupFilter.doFilter(SetupFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFil terChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain .java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java: 213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java: 178) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:10 7) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConn ection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:5 27) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorke rThread.java:80) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:6 84) at java.lang.Thread.run(Thread.java:595)

12

I just wanted to point out that this is still happening. It just happened right now.

13

I am having the same problem. Two of my buttons work but the third doesn’t. Has this been resolved yet?