We have a scenario where Openfire is used as the middle-ware for a distributed system. Recently, there were issues with Openfire throwing OutOfMemoryErrors and basically grinding to a halt. After some investigation, we found a script that was essentially sending large amounts of data (hundreds of MB) to every online user. The whole thing looked a bit like a Denial of Service attack.
We solved this by patching Openfire to throttle client connections using MINA’s ReadThrottleFilterBuilder. This filter limits the amount of data that can be read from the socket and waiting to be processed. As messages get processed, more data can be read from the socket again. The result is that it is no longer possible for a single sender to overwhelm Openfire and the server stays operational as if nothing happened. Normal clients, manually typing messages, would never trigger the filter.
I would rather not have to maintain a private branch with my patches and can submit the patch and a load tester application that simulates the problem to the Openfire developers. Would you be interested in applying this to Openfire?
Almost forgot: the problem was seen with Openfire 3.7.1 and still exists in 3.9.3. The patch should apply cleanly to the Openfire 3.9.3 source as distributed on the download page.
2014.08.05 20:31:59 org.jivesoftware.openfire.net.StanzaHandler - Error while negotiating TLS
java.lang.IllegalArgumentException: Unknown filter name:org.apache.mina.common.ExecutorThreadModel
at org.apache.mina.common.support.AbstractIoFilterChain.checkOldName(AbstractIoFilterChain.java:212)
at org.apache.mina.common.support.AbstractIoFilterChain.addBefore(AbstractIoFilterChain.java:125)
at org.jivesoftware.openfire.nio.NIOConnection.startTLS(NIOConnection.java:376)
at org.jivesoftware.openfire.net.ClientStanzaHandler.startTLS(ClientStanzaHandler.java:118)
at org.jivesoftware.openfire.net.StanzaHandler.negotiateTLS(StanzaHandler.java:403)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:161)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:181)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimpleProtocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:185)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.ReadThrottleFilterBuilder$Release.messageReceived(ReadThrottleFilterBuilder.java:186)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
The problem was at org.jivesoftware.openfire.nio.NIOConnection.startTLS(NIOConnection.java:376)
On this line, the SSLFilter is inserted right before a filter named org.apache.mina.common.ExecutorThreadModel. The filter in question however is in the chain under the name threadModel and thus can not be found. MINA’s ReadThrottleFilterBuilder documentation stipulates that its additions to the filter chain MUST BE right before and right after the ExecutorThreadModel filter, so we have to modify the filter chain by adding the SSLFilter before the filter added by ReadThrottleFilterBuilder.
It appears I am not able to add another patch, so I put it on pastebin. I’ve re-tested both with and without SSL and it all seems to work fine now.
BTW, the project summary in JIRA instructed me to file the bug-report here.
Bumping this thread as your last message contained a link and had to be approved. Usually no email notifications are sent after that. So making sure Dele will see that.
I think it is good to post a thread in the forums and then maybe attach a git request URL instead of attaching patches, so it can be discussed here instead of having eveything only in the comments on Github. Guidance is old and we have just recently moved to Git repository which allows for a different workflow.
Still not working with latest code on github. Does not like filter name "org.apache.mina.filter.ReadThrottleFilterBuilder.add"
2014.08.06 15:54:39 org.jivesoftware.openfire.net.StanzaHandler - Error while negotiating TLS
java.lang.IllegalArgumentException: Unknown filter name:org.apache.mina.filter.ReadThrottleFilterBuilder.add
at org.apache.mina.common.support.AbstractIoFilterChain.checkOldName(AbstractIoFilterChain.java:212)
at org.apache.mina.common.support.AbstractIoFilterChain.addBefore(AbstractIoFilterChain.java:125)
at org.jivesoftware.openfire.nio.NIOConnection.startTLS(NIOConnection.java:377)
Since you applied them on the github head, can you please give me a pull request instead. I am running out of time doing this and it will make much easier to apply this patch and close this issue.
I have pulled and merged your changes on github, pulled them into my local copy, rebuilt and was able to sucessfully log into Spark with both plain and TLS connections.