The future of Smack development: Google Summer of Code Project - Target Android and JavaSE

In the last few years, Smack development, has gained again some

momentum. I am very happy about this, but it’s not where it could and

should be. I’d like to start a discussion about future Smack and

aSmack, the Android port of Smack, development.

The current state of Smack development makes it not easy to get

patches originated from the Smack or aSmack community into a good

shape and finally into Smack. In the past, there was a lot of code,

and hence interested in Smack, that didn’t made it into Smack. Part of

which is due the differences of the two target platforms. It’s time

that the development of aSmack and Smack is unified again, reducing

the effort for aSmack and also giving more ressources to Smack. After

all, aSmack is the driving force behind Smack development, given the

fact that nearly every contribution to Smack of the last 3 years

orignated from aSmack.

A modular, modern build-system based, Smack, that targets Android and

Java, should be our goal. Modular in a sense that the user can select

the modules he needs, for example Core-IM with Service Discovery,

XHTML-IM and MUC, nothing else. Merging aSmack back into Smack,

unifying the build process, would be a win-win situation for every

single Smack user.

Google Summer of Code 2014 is coming, and I’d like to take that as

chance to get the project going. There are two important issues,

namely the development process and the build system, that we need to

tackle in two phases.

First, Smack’s VCS needs to be migrated to git. Ideally we would

release Smack 4.0 before that is done, so that the migration only

needs to take care of a single branch, ‘trunk’ in that case. Also a

proper review process needs to be established: At least another

developer should review commits before they get committed into the

master branch.

If we get accepted, modernizing and modularizing the build system will

be part of a GSOC 2014 project. Application starts 2014-Feb-03. You

can find the current wording of the project idea in DOC-2678.

So, I’d like to ask you, the Smack community for input about this. But

mostly I’d like to hear the opinion of my fellow Smack developer Robin

Collier about his thoughts.

2 Likes

First of all, I should mention that I am not too familiar with the current development situation of Smack. That being said, I welcome consolidating Smack and aSmack, as they seem to have very similar goals.

I would like to stress the usefulness and accessibility of having a repository on github or a similar system:

As for myself, my interest in Smack was triggered by the desire to have (mobile) Jabber client which wastes less bandwidth. Both yaxim and xabber use aSmack, so I set out to implement roster versioning support (XEP-0237). In the IRC channel, Flow pointed me to the aSmack repository, from which I extracted some old patches for the same future and rewrote them. For some months, I did not get around to to write a testsuite, so my patches didn’t go anywhere (which is good). Returning from my hiatus, I found there was a second attempt. As these patches were also on Github, it was easy to locate them and Flow’s review and integrate them in my (now hopefully finished and ready to review) patch.

In this process, I found also some places in Smack which I think deserve improving and would be willing to spend some time with that.

1 Like

I too would love to see all igniterealtime.org projects migrated to Git and perhaps to Github… it’s better is so many ways, there’s no point in trying to enumerate the benefits.

I’ve seen two problem here though.

  • First is manpower - we’re very short on manpower and the ones who are contributing have only a small amount of time. Moving over to Git is a big move which would require a lot of administration to get going. Things like svn2git make this reletively painless though.

  • Second is stubborness - we have a lot of seasoned and old-school programmers in the community who “grew up” using svn and cvs, etc… so a distributed scm is foreign to them and that makes some uncomfortable. Open Source projects are inheriently distributed in nature, so a distributed scm is only natural.

I’ve had a few of the igniterealtime.org repos mirrored under my github account, and actually have had some individuals engage with me simply because they stumbled upon it while browsing github. That’s Github though, not just git. Git alone will have many benefits and make it easier for people to just jump in and contribute.

A big secret that many might not know – almost all SVN commands will work on Github. As-in you can use SVN to administer and work with Github git repos.Github does magic in the backend and translates between the two programs. Pretty cool and surely will help companies and individuals migrate to using Git.

First is manpower - we’re very short on manpower and the ones who are contributing have only a small amount of time. Moving over to Git is a big move which would require a lot of administration to get going. Things like svn2git make this reletively painless though.

Manpower for the migration is no problem. In fact it’s pretty simple. If the offical git repo for Smack is setup, the plain migration is a matter of simply pushing smack there, as I already have Smack in git by using git’s svn functinoality. I therefore also see no need to use specialized tools like svn2git.

Connecting Bamboo and JIRA to the git repo is also possible. But as I don’t have the according rights, only Daryl can do that. But JIRA and Bamboo is not my main concern at this stage.

For what its worth, I have no objections with github and would be happy to modify infrastructure as needed to support it.

I’ve been working with Smack constantly since the last 2+ years. There are many things I dislike about it, but of course also a lot of things I like. But the most annoying thing is to work with XmlPullParser, in order to write new extensions. So maybe you can improve that.

I don’t understand the problematic of Android, i.e. why did it need its own build/branch in the first place? Is the Android Java only a Oracle Java 7 “light” without common SASL mechanisms and other things, like (Standard Java) XML parsing?

Could you explain in more detail please? (I’ve never worked with Android).

Concerning the BOSH branch: I am using it and did some improvements to it, like reconnection on exception and Smack 3.3.* compatibility. I could contribute the changes.

I’ve been working with Smack constantly since the last 2+ years. There are many things I dislike about it, but of course also a lot of things I like. But the most annoying thing is to work with XmlPullParser, in order to write new extensions. So maybe you can improve that.
The XPP interface is set in Smack and it’s unlikely that it will be replaced with a different XML parsing technique. What’s going to be modularized is the concrete implemention of the XPP interface, because of the different target platform come with different concrete implementations. And of course, one can always use the MXP1: XPP3 as Smack currently does.

On a side node: I believe that Android uses the same library as Samck, they just have different names for the conrecte implementation. That’s why it’s important that Smack uses an interface like XmlPullParser  |  Android Developers for parsing.

I am curious: What do you find anoying about working with XPP? It’s pretty straightforward and as I see it, the ideal way to parse XML received over a network stream. But if there is a better alternative, then please let me know.

I don’t understand the problematic of Android, i.e. why did it need its own build/branch in the first place? Is the Android Java only a Oracle Java 7 “light” without common SASL mechanisms and other things, like (Standard Java) XML parsing?

Could you explain in more detail please? (I’ve never worked with Android).

Android, or better the Dalvik VM and it’s API/libraries, is not Oracle Java. That’s it. IIRC they don’t pay license fee’s or so, and implemented everything from scratch while keeping the API of the implemented components compatible with Java SE. I guess the reasoning is that you don’t want a full featured Java API loaded into the VM on a mobile device, when most apps use only a small subset of the API. More information about the differences between aSmack and Smack are explained here: android - What is the difference between Smack and aSmack? - Stack Overflow and-asmack

Concerning the BOSH branch: I am using it and did some improvements to it, like reconnection on exception and Smack 3.3.* compatibility. I could contribute the changes.

That’s a good example why we need to migrate to git and finally merge the bosh branch. Many modifications by users don’t make it back upstream. You contribution is definelty welcomed, but I think it’s best to make one step at a time. So first we need to become clear about the future of Smack developmet.

The XPP interface is set in Smack and it’s unlikely that it will be replaced with a different XML parsing technique. What’s going to be modularized is the concrete implemention of the XPP interface, because of the different target platform come with different concrete implementations. And of course, one can always use the MXP1: XPP3 as Smack currently does.

So basically all what needs to be done is to write a simple factory which returns a XmlPullParser implementation, depending on the platform?

I am curious: What do you find anoying about working with XPP? It’s pretty straightforward and as I see it, the ideal way to parse XML received over a network stream. But if there is a better alternative, then please let me know.

It’s just cumbersome, error prone and hard to do it right, imo. Just have a look at org.jivesoftware.smack.provider.PrivacyProvider, how much code was needed.

I once implemented XML-RPC (xep-0009) and it was no simple task, especially due to the recursive nature of the XML structure.

In comparison JAXB is so much simpler to code. But this is probably not an option. But I guess even interfaces like javax.xml.stream.XmlEventReader are simpler to use, because you can deal with whole “XML elements” (e.g. xmlEvent.asStartElement()), which AFAIK XmlPullParser can’t.

But that’s probably not an option, too since it would break the whole library.

The XPP interface is set in Smack and it’s unlikely that it will be replaced with a different XML parsing technique. What’s going to be modularized is the concrete implemention of the XPP interface, because of the different target platform come with different concrete implementations. And of course, one can always use the MXP1: XPP3 as Smack currently does.

So basically all what needs to be done is to write a simple factory which returns a XmlPullParser implementation, depending on the platform?

That’s a part of one step in order to merge aSmack and Smack. The factory alone isn’t very useful, you need to refactor all classes, mainly the providers, to use that factory. The dependency injection needs to be designed and implemented, so that the Factory always returns the best available implementation. Plus there may be implementations that violate the contract, as can be seen in https://github.com/Flowdalic/asmack/issues/3.

Most of this sounds pretty good, and much of it fits in with my own vision of where I would like to see Smack go, including making Smack Android compatible. So it is good see that we are on the same page for most, if not all of this. I am going to address most of your points, but I am going to leave the VCS discussion for a different post, as IMHO this is something that should be an Igniterealtime wide initiative and I would like to keep this discussion Smack specific.

The modular build is a nice idea, I was considering doing this with a move to Maven, but the more I thought about it the more I rejected the idea. That is not to say that I am against the idea, but my own train of thought went from what you are proposing to thinking that it would be more complex than necessary since so many of the extensions in Smack are quite small. Due to this, I didn’t think there was much point to having a dozen different jar files with a possibly a large set of complex interdependencies. I am of the opinion that Smack is not really that large (compared to something like Sping for example) so in fact a single jar file is more than adequate for the majority of use cases. Exceptions to this would be any extension/capability that introduced 3rd party dependencies, such as an Android specific jar or Jingle. That being said, maybe there are advantages such as a smaller footprint that do outweigh that extra complexity. This is definitely something that can and should be discussed. As a side note, I would also have a separate module(s) for anything that has not reached the Draft state of the XEP process (as I mentioned once before), which would mean, for example, pulling out the existing workgroup implementation.

Another factor affecting this is that the way different functionality is enabled in Smack has to be changed. This would be part of the redesign of Smack which would eliminate the current practice of one size fits all for every connection. Once this practice is removed (static initialization that makes classes self enable), then having unloaded classes in a jar file really has no more affect than taking up some disk space.

Now, here is the problem with my vision. I see it as a rewrite of Smack. I think it is a very necessary one to change the design of Smack to make it much more flexible and usable in different environments in which it is currently used, but not well suited. It should be much more configurable, with sensible defaults that allow developers to configure each connection any way they want with the ability to share resources across those connections. Frankly, I am surprised that someone hasn’t already done what I have in mind already and effectively made Smack obsolete.

What I am thinking is that Smack 4 would pretty much be completely incompatible with lower versions. This would mean continuing to support 3.x versions for the next year or two, ideally strictly in maintenance mode once a reasonable first version of 4 is in place. I think, given the timelines, this means that version 4 could also be coded to Java 7.

That being said, I am not sure about waiting for a GSOC project for most of this as I would like to try to switch to Maven for 3.4. I do like your suggestion about the common pull parser API, that might make a nice little project. I was wondering if that would be possible because I was thinking that moving Smack 4 to Stax might be a good idea just to remove the external dependency on the pull parser. I haven’t inverstigated this though to determine whether that is actually a good idea, as I have never used Stax and certainly much more code could be reused from the current Smack by using the current pull parser.

I notice that you are proposing Gradle as a build system instead of Maven, I know very little about Gradle, so I am wondering what the advantage is of this over Maven in the case of Smack. My preference right now would be Maven for three reasons,

  • it is much better known (obviously due to its maturity)
  • it has excellent IDE support (this is a big one in my opinion)
  • I know it

I will post my thoughts on VCS and more importantly the overal workflow as soon as I am able. I am hoping to be able to do that tonight, but it may take a day or two.

By the way, my apologies for being so late to this discussion, I have family visiting these past 2 weeks and I am swamped and getting behind at work.

(If only I got paid for this instead)

Now, here is the problem with my vision. I see it as a rewrite of Smack. I think it is a very necessary one to change the design of Smack to make it much more flexible and usable in different environments in which it is currently used, but not well suited. It should be much more configurable, with sensible defaults that allow developers to configure each connection any way they want with the ability to share resources across those connections. Frankly, I am surprised that someone hasn’t already done what I have in mind already and effectively made Smack obsolete.

What I am thinking is that Smack 4 would pretty much be completely incompatible with lower versions. This would mean continuing to support 3.x versions for the next year or two, ideally strictly in maintenance mode once a reasonable first version of 4 is in place. I think, given the timelines, this means that version 4 could also be coded to Java 7.

Sounds interesting for me. Do you think a complete rewrite is reasonable and feasable?

How impactful do you think this rewrite could or should be in your vision?

Besides the configurability, which you mentioned, there are imo other basic design flaws, which could only be addressed with a complete rewrite. To name a few:

  • there is no class to represent a JID. Working only with String is error prone. Our code is full of StringUtils.parseName() and you are never really sure if your String is a JID or the username (localpart), especially in method parameters.
  • multiple “duplicate” classes for which a Java class already exists (HostAddress => InetSocketAddress, ProxyInfo => java.net.Proxy, Base64 => DatatypeConverter.parseBase64Binary(), SASLMechanism => javax.security.sasl.SaslClient, not sure though about the last one)
  • the XmlPullParser dependency. As said, the API is very inconvenient and just feels outdated.
  • Stream and Stanza errors are represented as String, that also makes it also impossible to retrieve the alternate address of the “see-other-host”, “gone” or “redirect” error and is also a little uncomfortable to work with (String comparison)
  • lower-case enum values (not conform with the Java Code convention)
  • maybe use Tinder.

Well, that probably expands on your vision…

That being said, I am not sure about waiting for a GSOC project for most of this as I would like to try to switch to Maven for 3.4. I do like your suggestion about the common pull parser API, that might make a nice little project. I was wondering if that would be possible because I was thinking that moving Smack 4 to Stax might be a good idea just to remove the external dependency on the pull parser. I haven’t inverstigated this though to determine whether that is actually a good idea, as I have never used Stax and certainly much more code could be reused from the current Smack by using the current pull parser.

I’ve made a little “proof-of-concept” XMPP Java project in my freetime, which uses Stax and JAXB. With Stax you mean the javax.xml.stream.* classes, right?

It works. Don’t know about Android though. But obviously this requires a complete rewrite, and I don’t know if it is really reasonable.

Actually Flow only asked for merging aSmack with Smack…

I am going to address most of your points, but I am going to leave the VCS discussion for a different post, as IMHO this is something that should be an Igniterealtime wide initiative and I would like to keep this discussion Smack specific.

There is no reason why this should become an igniterealtime wide initative. Most other projects besides openfire and spark are dead. And Gus, the openfire dev, already stated that he doesn’t want git.

The modular build is a nice idea… That being said, maybe there are advantages such as a smaller footprint that do outweigh that extra complexity.

It’s not only the smaller footprint. If a user has the option to not use components he wouldn’t suffer from their defects (e.g. because they have memory leaks).

This is definitely something that can and should be discussed.

Creating a modular Smack is a core objective of the GSOC proposal.

As a side note, I would also have a separate module(s) for anything that has not reached the Draft state of the XEP process (as I mentioned once before), which would mean, for example, pulling out the existing workgroup implementation.

The idea is to create a module for every XEP. And those modules carry metadata, such as their current status and type.

I was thinking that moving Smack 4 to Stax might be a good idea just to remove the external dependency on the pull parser. I haven’t inverstigated this though to determine whether that is actually a good idea

No, it’s not an good idea. We need an abstract parser that can be imlemented by the 3 different pull parsing implementations: XPP3 for non Java7, Stax for Java7 and MXParser on Android. That’s the other core objective of the GSOC proposal.

I notice that you are proposing Gradle as a build system instead of Maven, I know very little about Gradle, so I am wondering what the advantage is of this over Maven in the case of Smack. My preference right now would be Maven for three reasons,

  • it is much better known (obviously due to its maturity)
  • it has excellent IDE support (this is a big one in my opinion)
  • I know it
    I think Gradles flexibility makes it better suited for the job and therefore I prefer it over Maven. Also. it is more powerful and supported by (most?) IDEs.

I see we have different ideas regarding the future of Smack. I think we need a modular build system provided by gradle and an abstraction for some components of Smack so that Smack can be deployed on various platforms.

You propose to remove the static initializers including of a rewrite of large parts of Smack and keep Smack monolithic for some reason.

I hope that’s an accurate summary of the situation.

The modular build is a nice idea… That being said, maybe there are advantages such as a smaller footprint that do outweigh that extra complexity.

It’s not only the smaller footprint. If a user has the option to not use components he wouldn’t suffer from their defects (e.g. because they have memory leaks).

In the design I am talking about, this would never occur as the class wouldn’t be loaded unless it was wanted anyway. This only occurs now because of the issues with static initiialization and the diffuculty of changing the built in defaults that are loaded from the embedded config file. This last part will soon be fixed anyway as I have a working version of a more flexible provider manager initializer. I still have some testing and documentation to write, but it will make it simpler to change Smack’s default configuration.

In either case, I am not against having the modules, as long as there are some consolidated modules to go along with them. I know I don’t want to have to pull in 20 jar files to enable the features I want to use when I can do it with a couple, especially when many of those jars will probably only have a couple of classes. It is easy to satisfy both cases though, it doesn’t have to be one or the other.

I was thinking that moving Smack 4 to Stax might be a good idea just to remove the external dependency on the pull parser. I haven’t inverstigated this though to determine whether that is actually a good idea

No, it’s not an good idea. We need an abstract parser that can be imlemented by the 3 different pull parsing implementations: XPP3 for non Java7, Stax for Java7 and MXParser on Android. That’s the other core objective of the GSOC proposal.

Could you elaborate on this a little more, please. Stax is availble since Java 6 and with a dependency also for Java 5.

Furthermore it is just an API, a set of interfaces, for which several implementations exist, like Codehaus, Woodstox, and so on.

In this article it says “Android embraces it”, so I guess there’s also an implementation for Android.

As I understood you, you want to introduce a new interface / abstract class, which acts as an adapter between XPP3, Stax and MXParser. But imo Stax is already the common API you are looking for, no?!

I know I don’t want to have to pull in 20 jar files to enable the features I want to use when I can do it with a couple

I also don’t want the outcome of a module to be jar file and should probalby made that point clear. I think of configurable sets of modules, where some configurations are pre-defined (full, minimal, minimal-chat, …) but can also be provided by the user, and gradle bakes the according jar based on the configuration (e.g. core-im with entity-caps and xhtml-im).

Stax is availble since Java 6 and with a dependency also for Java 5.

Correct, my fault. So it’s stax (in particular the interface XMLStreamReader) for everything from Java6 on.

Furthermore it is just an API, a set of interfaces,
IIRC Stax defines interfaces and concrete implementations (i.e. XMLInputFactory.createXMLStreamReader(InputStream) does return an pull parser). Woodstox is part of Codehaus, implements the Stax interface but is not shipped with Java SE. I really like to keep the dependencies minimal, but that doesn’t prevent anyone from creating a module that provides woodstox for xml parsing.

In this article it says “Android embraces it”, so I guess there’s also an implementation for Android.

No, there is none. Search Package Index  |  Android Developers for the javax.xml.stream namespace. It does not exist (for the reasons mentioned earlier).

As I understood you, you want to introduce a new interface / abstract class, which acts as an adapter between XPP3, Stax and MXParser.
Correct, that’s beside a modular build system, the most important goal for Smacks future. And a Smack that targets Android and Java SE demands this, which then also means that we need a modular build system.

But imo Stax is already the common API you are looking for, no?!

Not if take Android into account.

No, there is none. Search http://developer.android.com/reference/packages.html for the javax.xml.stream namespace. It does not exist (for the reasons mentioned earlier).

Ok, right. I googled it up. Seems like you have to use XmlPullParser for Android.

But in this case I don’t know if it is a good idea to introduce a new set of interfaces just to be used as an adapter between XmlPullParser and StAX.

Just keep using XmlPullParser and use a factory, something like this:

public class XmlPullParserFactory()

{

public XmlPullParser newInstance()

{

// Check if Android dependency is on the classpath, if yes:

return Xml.newPullParser();

else

{

return new MXParser();

}

}

}

But in this case I don’t know if it is a good idea to introduce a new set of interfaces just to be used as an adapter between XmlPullParser and StAX.

We are not introducing a set of interfaces. Just one, and the platforms pull parse implement it. And I don’t see a problem with it.

If you would go with your suggestion, then we would add an unnecessary dependency because Smack would depend on MXParser on >= Java6, when it could simply use the Java SE provided Stax parser.

Ah ok, I understand. But maybe it is still possible to use the Java StAX interface(s), even on Android. Maybe it is possible to write an adapter for a XmlPullParser, i.e. a class which implements XMLStreamReader and gets an XmlPullParser in its constructor and then delegates the calls to the XmlPullParser.

I only wonder why nobody else has done this yet, but I can’t find anything like that, maybe it’s not as trivial as I think.

The advantage would be, that you use a standard interface instead of introducing your own. And maybe Android will support StAX someday, too, so your interface would become obsolete.

Maybe it is possible to write an adapter for a XmlPullParser, i.e. a class which implements XMLStreamReader and gets an XmlPullParser in its constructor and then delegates the calls to the XmlPullParser.

I am sorry, but I don’t understand this approach nor the advantages.

The advantage would be, that you use a standard interface instead of introducing your own

The interface we create in Smack will be equal to the standard XPP/Stax interface. We can’t reuse the “standard” interface because it’s not available on every platform. We are talking here about a simple interface .java file, that would look exactly the same as the XPP and Stax interface.

And maybe Android will support StAX someday, too, so your interface would become obsolete.

I hope that Android will never support natively stax. It would just increase memory consumption, which is not a good idea on embedded devices. The Android API should only be kept minimal. After all if you need something, just add it to your app (Which is actually the recommended procedure for Android applications that need javax.* or other stuff not natively provided).

Also even if Android would one day support stax, we still need to be backward compatible with older Android versions.