Results of test are very disconcerting!
The 100 sending clients and server were run on the same machine. The receiver on another. Initially, the senders and server were run at the same priority. The senders run as fast as they can with no delay between message sends. A main thread dispatches the commands to the senders to send the messages so the test is somewhat serialized. Sometimes (twice) 1 million messages were sent and received. But in most runs an exception would be thrown in sender and it would die. The exception would typically be something like:
No delay sent: 948639 received: 156165
Apparently, a sender died for unknown reasons, no error message, no sign of socket closing?
java.net.SocketException: Software caused connection abort: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder$CharsetSD.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.fill(Unknown Source)
at java.io.BufferedReader.read1(Unknown Source)
at java.io.BufferedReader.read(Unknown Source)
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(Unknown Source)
at com.sun.net.ssl.internal.ssl.OutputRecord.write(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(Unknown Source)
at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(Unknown Source)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(Unknown Source)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(Unknown Source)
at sun.nio.cs.StreamEncoder.flush(Unknown Source)
at java.io.OutputStreamWriter.flush(Unknown Source)
at java.io.BufferedWriter.flush(Unknown Source)
Exception in thread “main” java.lang.IllegalStateException: Not connected to server.
The server logs are empty or indicate a comm failure. The results were not very repeatable and show no pattern.
I increased the server priority to above normal and behaviour changed. No more exceptions!
But, there is always a but.
30% of the messages never make it to receiver! There are no errors anywhere indicating anything went wrong!
If you take I-75 North out of Atlanta (7 lanes) it eventually goes to two lanes. Usually traffic backs up painfully and it can take an hour to go a few miles. So one would expect in a backed up fully loaded scenario to eventually get to destination.
Instead the choices become:
For no good reason you are involved in a crash.
You eventually make it.
You dissappear mysteriously from the face of the Earth never to be seen again and noone knows why.
Turning on server debug yields no clues. I pretty much out of ideas on what to do but will continue to test and think about as I have time.
What this tells me though is there is no guarentee your sent message will be received in a heavy loaded situation. I believe there is a problem here somewhere altough most users may never experience it.