File Transfer Bytestream Failure

We’re currently diagnosing a bug seperate from this topic. The symptoms are that out of band fails and in band occurs. We are using XMPP to transfer lage amounts of files (its complicated). However, we noticed that once OOB trips then EVERYTHING is ALWAYS OOB. This appears to be by design as I understand the disableService calls:

http://fisheye.igniterealtime.org/browse/smack/trunk/source/org/jivesoftware/sma ckx/bytestreams/socks5/Socks5BytestreamManager.java?hb=true

http://fisheye.igniterealtime.org/browse/smack/trunk/source/org/jivesoftware/sma ckx/bytestreams/ibb/InBandBytestreamManager.java?r=11821

Am I reading these correctly? Why the assumption that every subsequent OOB fails? Aren’t there conditions where the server is busy and a legit failure occurs that doesn’t bear on future transmissions? I suppose the workaround is to add the service back every time that an in band finishes in my client code.

Hi Nicolas,

you are on the same spot as we are. There are several issues with Smack 3.2. First of all the code introduces by Henning in r11821 was changing the default stanza for IBB to IQ instead of the formerly used message stanza. The IQ stanza are failing in Spark hence we move back to message stanza in the Smack 3.2.2 alpha. See Smack-351 for the patch associated.

The second thing that is most disturbing is that Bytestreams and IBB are sometimes working and sometimes not. If you have a patch or anything else that help tracking this down is highly welcome. And you are not alone…

I saw you did post some other stuff related to Smack file transfers and you were also refering to a larger project with releases.

Since you are working persumably on the Smack 3.2. branch: Any patches or corrections your project has done to Smack that you want to share?

Did you get IBB working with IQ stanzas? We are have real big issues in Spark and move back to message stanzas in the next Smack release.

We’ve noted two issues. First that we lose byte stream if anyting “bad” happens. I think that’s identified by the code that I linked to. I’m not keen on removing it because there might be good reasons to drop out of band traffic. I’m too new to know what those might be. I’m not sure though that the assumption of in-band always being long term preferable is right.

Second, and there is a thread elsewhere from (I think) '09 about a thread leak. When we leave these up for hours at a time we eventually cap our allowed threads (10k). Some of these are ours, but only about a hundred of them. I haven’t had time yet to cross reference the '09 post to the current code.

Since we only use Smack I haven’t noticed any problems with the stanza type I imagine since the clients on both sides are symetric. In fact I knew nothing about htis till you mentioned it.

Both of the problems are bad since we are using Smack for a “back-end” process and didn’t run long tests so didn’t see these errors. They only manifest after several hours of heavy usage (10k+ file transfers of 200k-3MB).

I should note that our general feeling is that Smack may have some long term problems with it and we’re actually preparing to wrap the whole thing up in a bootstrapper that has a watchdog. Something crazy happened? Kill it, start over, and move on. If we can get Hennering or whoever made the change to comment here I can propose a patch for the titled problem.

Just to get everything cleared: What is the Smack revision / branch you are using? If it is 3.2.1 and IQ stanzas are working for you, we have to review the Spark code.

This respect to Henning, who did the file transfer redesign, he is not with the project anymore. Unfortunately.

Kind regards

Walter

I started looking at the code last night and I am hoping to get up to speed on how it works (or is supposed to work) over the next week. I started looking at some of the relevant specs last night as well and I expect a rather large learning curve as I have never looked at the code or feature before.

Hopefully we can get it all sorted out soon enough (including the threading issue) so that it is fixed once and for all. I have to go take a look at what Henning was fixing/enhancing as well to figure out what the changes were for.

hoping to have more time for this after Saturday. I have my daughters birthday party tomorrow and a demo for work on Monday (work I acctually get paid to do ).

There are apparently at least two issues that plague Spark2.6.3/Smack3.2.1:

IQ stanza for IBB transfers are failing

The negotiation of the network connection during the initiation of Bytestreams fails and Smack reverts to IBB. This is happening even though 7777 connections are possible. It might be a pretty low level issue. Tim is working on this.

What is missing in the moment is a confirmation that Smack 3.2.1 can do file transfers via IBB/IQ Stanzas, because this would point to an issue in the Spark code. Currently the Smack alpha reverts to message stanzas to allow IBB file transfers in Spark 2.7.0. But this is actually a bad idea since IQ stanzas seem to be the more standard conformant IBB implementation.

So I want to put forward the possibility that the out of band service removal is done on a timer. That is, after five minutes or some other settable time expires then the service is re-added and future transfers will try to use out of band.

Also, we’ve got a test suite set up to help us track down the threading issue. We’ll try to report back more as we dig into it this week. This is unforunately ancilliary so it might be slow.