Packet Storm on 3.5.1

Openfire Openfire Support
1

Am experiencing a packet storm on our OpenFire 3.5.1 server. We currently use LDAP accessing Active Directory and use a SQL Server database. Here is what is in the error log:

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:180)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)
2008.08.28 13:52:39 [org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:69)
] Internal server error
java.lang.NullPointerException: replacement
at java.util.regex.Matcher.replaceFirst(Unknown Source)
at java.lang.String.replaceFirst(Unknown Source)
at org.jivesoftware.openfire.ldap.LdapVCardProvider$VCard.treeWalk(LdapVCardProvid er.java:521)
at org.jivesoftware.openfire.ldap.LdapVCardProvider$VCard.treeWalk(LdapVCardProvid er.java:525)
at org.jivesoftware.openfire.ldap.LdapVCardProvider$VCard.getVCard(LdapVCardProvid er.java:502)
at org.jivesoftware.openfire.ldap.LdapVCardProvider.loadVCard(LdapVCardProvider.ja va:216)
at org.jivesoftware.openfire.vcard.VCardManager.getOrLoadVCard(VCardManager.java:2 05)
at org.jivesoftware.openfire.vcard.VCardManager.getVCard(VCardManager.java:198)
at org.jivesoftware.openfire.handler.IQvCardHandler.handleIQ(IQvCardHandler.java:1 08)
at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:49)
at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:349)
at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:101)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)
at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:299)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:264)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:163)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:180)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)

Currently there are 114 users and only 6 conversations active.

Any idea what is happening?

2

I have been having the same issue (http://www.igniterealtime.org/community/thread/34646?tstart=30).

From other threads it appeared to be an issue with random JIDs trying to hit sub-domains of others servers, but from my logs it looks like it could be a presence caching issue. Maybe there are two issues with the same symptom?

3.6.0 subversions were noted to have solved the sub-domain and server-to-server issues, and once 3.6.0a is released we’ll see if that does or does not solve the packet storms in general.

3

Did you ever get a resolution to your packet spraying/storm issue? I’m still having mine and it appeared to start when some users started trying the Exodus client. Don’t know for sure if it’s related but we did tei one of the recent packet storms back to one of the Exodus users. It also appears to me to be a deluge of presence related packets.

4

What means packet storm?

Some time ago I experienced an issue where an single packet endless looped inside Openfire. This produced over 7,000 packets per second. I never found out what was happend, but it was easy to solve. I needed just to add an explict drop rule for that packet.

Since the Raptor plugin is public, such things are relativly easy to do…e.g. you could use a Raptor profile like this to find out which user is causing your problem.

(Raptor does require at least Openfire 3.5.2)

<?xml version="1.0" encoding="UTF-8"?> <raptor xmlns:action="http://martin-weusten.de/raptor/action" xmlns:check="http://martin-weusten.de/raptor/check">
    <name>Raptor01</name>
    <desc>This example detects and logs users causing much traffic. They are logged if they produce more than 33.3 packets per second over longer time.</desc>
    <version>1</version>
    <counter name="TRAFFIC" interval="3000" decrement="100"/>
    <action:branch>
        <rule>
            <then>
                <action:count counter="TRAFFIC" count="FROM"/>
            </then>
        </rule>
        <rule>
            <if>
                <check:count counter="TRAFFIC" count="FROM" compare="GREATER" ref="2000"/>
            </if>
            <then>
                <action:set_count counter="TRAFFIC" count="FROM" newvalue="0"/>
                <action:log>Much traffic from user '\F' @ '\I', packet:\X</action:log>
            </then>
        </rule>
    </action:branch>
</raptor>

Message was edited by: Coolcat

5

By packet storm I mean a sudden inexplicable jump in number of packets per minute as shown in the statistics page in the admin console (have monitoring plugin active). We normally run below 500 PPM. However, yesterday I saw 108,000 PPM. In the statistics page it shows as dark blue in the graph so I suspect its presence & IQ packets. We usually have to restart the OpenFire service to clear the storm.

Is there a zip version of the raptor plugin download package?

Jerry Zeiszler

6

Is there a zip version of the raptor plugin download package?
Now it is

7

I’m having the same packet storm, with users who are using exodus. I think there is an issue with trying to create drop rules everytime this happens. It seems to occur on random users, and it almost never is the same user. In my environment, where about 400 users are using exodus, I’d have to camp out on the statistics page waiting for the packet storm to happen and create a new drop rule every time. Honestly, it’s not a functional problem unless packet auditing is enabled - if packet auditing is enable openfire will try to write every one of those 100k packets per minute to log files and drag the server to it’s knees, where users are no longer able to send messages because the system is simply too busy.

The message archive plugin is not effected by the packet storms.

8

Do you get the same log file error messages that I put in the attached file (i.e. Username serveralert not found)

Jerry Zeiszler

9

Could you both try to log that packet that is looped using Raptor? The profile I posted above will do that. I will only log approx. one of 2000 packets, so don’t worry about log size.

10

I can, but it won’t be immediately. The Raptor page notes that the plugin is in alpha, so I can’t load it up quite yet. Since I wouldn’t have enough users to reliably create a packet storm in my test environment, I don’t believe that would be a good test…

What I will try to do is get this going on an upcomming weekend (when my users can deal with broken IM if something goes horribly wrong) just to be on the extra safe side.

11

Yes, that plugin is still alpha…because I expected some bugs there. However, I’m using this thing myself on my production system for weeks now with exactly that profile I posted here. Also nobody else reported bugs to me.

I think you have heavy problems, using the plugin it can’t get worse

12

Yeah, I completely agree with you. The way plugins work in openfire it’s easy enough to back out even if we did run into issues… Still, processes to follow for change management and whatnot.