Problems with manifest files in 4.0.0-rc2 regarding OSGi

Smack Smack Dev
1

Hi Flow,

since the projects’ renaming there is a problem in the manifest files.

smack-core:

“Bundle-SymbolicName” appears twice; one is “org.igniterealtime.smack.core” and the other is “org.igniterealtime.smack”. Please be aware that all the fragment bundles carry “org.igniterealtime.smack”. There is an entry “Name: Eclipse-ExtensibleAPI: true”. It must be “Eclipse-ExtensibleAPI: true”.

smack-x:

The “Bundle-SymbolicName” of all the fragments is e.g. “org.igniterealtime.smack-smack-debug”. I think it should be “org.igniterealtime.smack-debug”

I will try to correct the problems manually, but it would be great, if you can fix it.

2

It appears that the Fragment-Host approach is no longer needed in Smack 4. From studying the previous discussions regarding OSGi and Smack it seems this required because smack (now smack-core) had a dependency on a file (smack.providers) in smackx (now mostly smack-extensions). This shouldn’t be the case with Smack 4 any more.

@Jens: Could you grab https://github.com/Flowdalic/Smack/commit/b42598e5ac8efdd3f0e281a7bf63482553d06d 3b and compile the jars for testing purposes? The commit removes those Fragment-Host and PatchFragment attributes from the manifest. The less extra configuration we have the better the maintainability of Smack will be. Would be great if you coud report back if Smack still works in an OSGi environment with this commit.

3

“smack-core” and “smack-tcp” both export the package “org.jivesoftware.smack”. In OSGi, this is called “Split Packages”. Have a look here: http://wiki.osgi.org/wiki/Split_Packages. It has to be explicitly declared in the manifest files, but it is best practice to avoid this at all. The best solution is to make sure that one package is only exported by one bundle.

4

I just made a quick expierment moving the tcp/bosh code into org.jivesoftware.smack.(tcp|bosh). I turns out we have a lot of visiblity problems to solve in order to move the tcp/bosh subproject code into their own package. But as I understand the wiki page, this is the recommended solution.

I will give it some more time to think about how this is best dealt with. Thanks for your valuable input Jens. BTW: Do you see any more problems regarding OSGi support of smack besides the mentioned “split package” situation?

5

Yes, the movement to different packages is the best solution. Split-packages should only be used if there is really no other possibility. Especially, in the OSGi world, you have to be very sensitive towards classloading and visibility issues. In my use case, I am not using the full potential of Smack and make only use of the PubSub-feature.

Currently, I would say that things become much better and enhance Smack’s OSGi compatibility a lot. I was not aware that you have dissolved the dependency between the core and the extensions. If it is possible to create a solution without the fragment approach, that would be preferable. But, please be aware of the consequences: Every bundle now has its own classloader, so every time you make use of Class.forName or of a classloader, you must be aware what classes are accessible and visible in the OSGi world. I think the golden rule to satisfy is, that there must be no direct or indirect dependency from the core to any other bundle. That was the problem in version 3.x.x, where the core had an indirect dependency to the extension bundle.

6

Could you grab the dev version from

https://github.com/Flowdalic/Smack/tree/osgi

and comment on the OSGi functionality? I’ve moved the TCP/BOSH code into their own packages, to the smack-(tcp|bosh) subprojects should not longer expoert org.jivesoftware.smack

I need to know if this fixes any OSGi related problems in Smack, so that I can proceed with preparing 4.0.0-rc2.

7

Your package movements work, but at first glance, there are still some issues to solve:

(1) Split-Package#1:

“smack-core”, “smack-resolver-javax” and “smack-resolver-dnsjava” all of them export the package “org.jivesoftware.smack.util.dns”.

(2) Versioning#1:

“smack-bosh” imports the package “org.igniterealtime.jbosh;version=”[0.8,1)" in a wrong version and exports it in the current version "org.jivesoftware.smack.bosh;version=“4.0.0.rc2-SNAPSHOT”. Is the self-import really necessary?

(3) Versioning#2:

“smack-resolver-javax” and “smack-resolver-dnsjava” import the package “org.jivesoftware.smack.util” without any version. It works, but it should be avoided.

8

Regarding 2: I think you confused org.jivesoftware.jbosh with org.jivesoftware.**smack. **The import looks correct to me.

Will fix 1 and 3. Thanks for your input. smack-compression-jzlib has likely the same split package problem as smack-resolver-*, will fix that too.

9

Regarding (2):

Oh yes, I am sorry. It is “org.igniterealtime.jbosh;version=”[0.8,1)".

“smack-compression-jzlib”:

Yes, you are right, I just found it, too. “smack-core” and “smack-compression-jzlib” both export the package “org.jivesoftware.smack.compression”.

10

New commit pushed. Should address all issues, but the one where some subprojects/packages import e.g. “org.jivesoftware.smack.util” without a version. Not sure if there is an easy fix for it, probably not. @Jens Would you consider this a blocker regarding Smack’s OSGi compatibility?

11

Oh Flow… We are in big trouble!

To answer your question first:

No, I don’t think this is a blocker. I am not familiar with gradle, but the gradle OSGi plugin uses BND like the OSGi Maven Plugin. During development of my project, I was faced with the same problem. In my case, there was always a problem in the Maven POMs. I think, the problem will be solved in the course of time.

But now I have a blocker:

I thought that you got rid of the implicit dependency between the core and the extension bundle. As far as I understand the code, this is not the case. With your current version I get the following exception:

  1. java.lang.ClassCastException: org.jivesoftware.smack.util.PacketParserUtils$UnparsedResultIQ cannot be cast to org.jivesoftware.smackx.disco.packet.DiscoverInfo

From my point of view, the providers from the extension bundle are not loaded. I cannot find a trigger inside the extension bundle. The reason why it worked before was because the extension bundle was a fragment and the providers were loaded by the core looking at “smack-config”. Now the core looks at the config file and tries to load the providers. That fails, because the classes are no longer visible to the core. We have four options:

(1) The extension bundle becomes a fragment again.

(2) We can add optional imports of all smackx-packages (very error-prone)

(3) We add “DynamicImport-Package” to the OSGi manifest (http://wiki.osgi.org/wiki/DynamicImport-Package, I do not recommend that, because of performance issues)

(4) We find a way to add the providers from the inside of the extension bundle.

Currently, I have no good idea for proposal (4), because we must find a way to initialize the providers first before any class from the extension bundle gets used… If you implement the interface “Bundle-Activator” and put the class in a specific package which is only accessed from OSGi, we could implement the init code. That’s not so nice, because the extension bundle has an explicit OSGi dependency. It still runs standalone, because anybody accesses the corresponding classes. At the moment, I would prefer (1). What is your opinion.

12
  1. java.lang.ClassCastException: org.jivesoftware.smack.util.PacketParserUtils$UnparsedResultIQ cannot be cast to org.jivesoftware.smackx.disco.packet.DiscoverInfo
    Would be great to know where in Smack this happens. Could you show us some more stacktrace?

From my point of view, the providers from the extension bundle are not loaded. I cannot find a trigger inside the extension bundle.
The process with Smack 4 is: SmackConfiguration → org.jivesoftware.smack/smack-config.xml → org.jivesoftware.smackx.ExtensionProviderInitializer → org.jivesoftware.smackx/extensions.providers

That fails, because the classes are no longer visible to the core.
Could you elaborate that? I assume the classloader of core is not able to load other classes outside of the core OSGi bundle. Is this correct? If so, is there any way to change that without resorting to using fragments? If not, I think the only sane option would be to use fragments again.

But I want to fully understand the problem first. So some pointers to OSGi wiki/documentaiton regarding this topic are appreciated.

13

This is the stacktrace:

java.lang.ClassCastException: org.jivesoftware.smack.util.PacketParserUtils$UnparsedResultIQ cannot be cast to org.jivesoftware.smackx.disco.packet.DiscoverInfo

at org.jivesoftware.smackx.pubsub.PubSubManager.getNode(PubSubManager.java:172)

…

at org.jivesoftware.smack.XMPPConnection$ListenerWrapper.notifyListener(XMPPConnec tion.java:1221)

at org.jivesoftware.smack.XMPPConnection$ListenerNotification.run(XMPPConnection.j ava:1131)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Every bundle (except fragments) has its own classloader. Via “Export-Package” and “Import-Package” classes can be shared between different bundles, but every class is linked to the classloader that has loaded the class for the first time.

Well… I have an idea. I have generated the eclipse settings and will try to import the projects to my eclipse. The approach is as follows:

At the very beginning, the user must call

new ExtensionsStartupClasses().initialize();

new ExtensionsProviderInitializer().initialize();

With the current version, this is currently not working and I get:

WARNING: No input stream created for classpath:org.jivesoftware.smackx/extensions.providers

We must pass the correct classloader to FileUtils… It is part of the signature but never used. I will try to modify the code locally and make a quick test.

14

I got it working without fragments… But I had to make some changes to the code. I only get this:

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.disco.ServiceDiscoveryManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.xhtmlim.XHTMLManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.muc.MultiUserChat’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.bytestreams.ibb.InBandBytestreamManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.bytestreams.socks5.Socks5BytestreamManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.filetransfer.FileTransferManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.iqlast.LastActivityManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.commands.AdHocCommandManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.ping.PingManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.privacy.PrivacyListManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.time.EntityTimeManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.SmackConfiguration loadSmackClass

WARNING: A startup class ‘org.jivesoftware.smackx.vcardtemp.VCardManager’ could not be loaded.

Mai 08, 2014 5:03:01 PM org.jivesoftware.smack.initializer.UrlProviderFileInitializer initialize

This is normal, because the corresponding classes are not visible to the core. From my point of view, it would be preferable to remove this implicit dependency between the core and the extension bundle. The trade-off is, that the user has to call something like “Extensions.initialize()” to trigger the provider addition.

What’s your opinon? Should we stay on the fragment approach or should we add an initializer to the extension bundle?

15

But I had to make some changes to the code.
Would you mind showing us a diff?

The trade-off is, that the user has to call something like “Extensions.initialize()” to trigger the provider addition. What’s your opinon?

That’s actually where we come from if you look at aSmack: User had to run a init() method before aSmack would work correctly. This caused a real terrible user experience. So no, this implicit dependency (over the optionalStartupClasses) will stay. It works for “normal” Java encironments and Android.

But we could still provide an initalizer for the OSGi users, or go with the fragment approach. I am undecided and would like to wait a few days. Maybe a third option appears (although it seems unlikely from what I know right now about OSGi).

16

I have created the most important Diffs and attached them to this answer. Ok… Let us allow the issue to rest some time. I understand your concerns regarding user experience. From my point of view the fragment approach is the most suitable one that’s solves our issues without changing the initialization behavior in OSGi based environments, too.

I have created a personal version for me and added it to my private maven repository in order not to interfere with my development progress. When rc2 is officially released I will switch to this version.

Thanks for your help!
classloading-diffs.zip (2962 Bytes)

17

After reading some more about OSGi, I was first happy that I found BundleActivator, which seemed to be exactly what we wanted for Smack’s initialization use-case. But then again, BundleActivator is a interface of the OSGi framework, which means that if we want to use it in Smack, we had to introduce a hard dependency on osgi-core, but this violates Smack’s minimalistic principle and also raises the question how to deal with Android.

In order to provide OSGi users a convinient way to initialize Smack’s extensions, experimental and legacy subproject/bundle, I refactored the initializers so that every bundle provides a single initialze() method (instead of two). The idea is that OSGi users now only have to call e.g. o.j.smack.initializer.ExperimentalInitializer.initialize() in order to init the Providers and static code of the smack-experimental bundle.

If only OSGi would fallback to using reflection for BundleActiviator while using more simple start() and stop() methods, namely ones without argument. Then we could simply point to the Initializers in the manifest and OSGi users wouldn’t have to deal with this at all.

@Jens: Do you happen to know a place where I can report OSGi feature requests? And, as always, would be great if you could grab my osgi branch of Smack and report back if it works out for you. Thanks.

18

I can only test your changes on Monday, because my local development environment here is not properly configured. Have a look to the class FileUtils, the classloader provided in the signature is never used. Can you please fix this?

Ok, than another approach. The thing you are looking for has already been implemented and specified by the OSGi Alliance. It is called “Declarative Services”, especially since version 1.2 a very powerful extension. It works as follows: You do not have to implement specific OSGi interfaces, the only thing that is required is an xml file located in the folder “OSGI-INF” in the bundle jar. This allows you to declare dependencies and offered services. At the moment we only need at startup that OSGi or more specific the “Service Component Runtime” (SCR) instantiates the class “ExtensionsInitializer” and calls “initialize()”. No problem. The file must look as follows:

<scr:component enabled=“true” immediate=“true” name=“org.jivesoftware.smack.initializer.extensions.ExtensionsInitializer” activate=“initialize”>

</scr:component>

What’s pretty cool, is that the file can be hardcoded or automatically generated. In Maven, there is the “Apache Felix Maven SCR Plugin”. It uses annotations that only exist at compile time. Declare a component and annotate the method “initialize” with @Activate. The xml file is generated and put to the jar. As I told you, I am not familiar with gradle, but I found this: https://github.com/janvolck/gradle-scr-plugin. Is it possible to integrate and test it? Have a look here: http://stackoverflow.com/questions/12642706/how-to-use-org-apache-felix-scr-anno tations-for-a-bundle-from-scratch. It is a simple example showing how the annotation based approach works.

19

I can only test your changes on Monday, because my local development environment here is not properly configured.
Sure, no hurry, we all need weekends

Have a look to the class FileUtils, the classloader provided in the signature is never used.
Ahh, good catch. Will fix this.

The thing you are looking for has already been implemented and specified by the OSGi Alliance. It is called “Declarative Services”,
Great! Will definelty look into it.

What’s pretty cool, is that the file can be hardcoded or automatically generated. In Maven, there is the “Apache Felix Maven SCR Plugin”. It uses annotations that only exist at compile time.
Those annotations are actually a problem: aSmack is build form sources, and the org.apache.felix.scr.annotations.Activate import will result in a compilation error if Smack is build against the android.jar. But nevertheless I think we are on a good track here. The XML file can simply be added as resource file. After all the felix annotations and the resulting automatic generation of the xml resource file provides very little benefit in Smack.

Will experiment a bit with it and let you know of the outcome.

20

I have checked out your modifications… Thanks for the update, but there is a problem. The files “*component.xml” MUST be contained in the folder named “OSGI-INF” in the root level of the jar and the entry in the manifest files must be e.g. “Service-Component: OSGI-INF/smack-extensions-components.xml”.