I have tried to migrate to Smack 4.1.0-alpha1 on OSGi, but unfortunately I receive a error that no SASL authentication methods are available… I have added the bundle “smack-sasl-javax” to my run configuration, but I was unable to get it working. There is one big problem caused by the fact that “smack-sasl-javax” is a stand-alone bundle. When “smack-core” resolves it is not guranteed that any SASL authentication method is available within Smack. It depends on an unpredictable activation order at startup which leads to the problem that “smack-sasl-javax” is activated later. If the core requires any SASL authentication there must be an explicit dependency otherwise things are not working. But even after getting Smack running, I got a “NoResponseException”, so that I was forced to downgrade to 4.0.4.
Thanks for the report Jens!
If I understand you correctly then the problem is caused by smack-core being activated before smack-sasl-javax, causing smack-core to not see the smack-sasl-javax components.
Do you have an idea on how to set the activation order?
I am not really sure… I have added a package-dependency to my bundles to force that during startup my stuff is resolved after Smack, but I got the following exception:
02:55:01 PM SENT (242792115): <stream:stream xmlns=‘jabber:client’ to=‘cloudv207.zdv.uni-mainz.de’ xmlns:stream=‘http://etherx.jabber.org/streams’ version=‘1.0’>
02:55:01 PM RCV (242792115): <?xml version='1.0' encoding='UTF-8'?><stream:stream xmlns:stream=“http://etherx.jabber.org/streams” xmlns=“jabber:client” from=“ec-xmpp” id=“3b1f23ab” xml:lang=“en” version=“1.0”>
02:55:01 PM RCV (242792115): stream:featuresDIGEST-MD5PLAINANONYMOUSCRAM-MD5</mech anism>zlib</stream:features>
02:55:01 PM SENT (242792115): =
02:55:01 PM RCV (242792115): cmVhbG09ImVjLXhtcHAiLG5vbmNlPSJyWWpCYj Znc1I4WUVKZXNZUXV2bXBFQWNoRVVGRGowTnJnUFcxRjFmIixxb3A9ImF1dGgiLGNoYXJzZXQ9dXRmLT gsYWxnb3JpdGhtPW1kNS1zZXNz
org.jivesoftware.smack.SmackException$NoResponseException
at org.jivesoftware.smack.SASLAuthentication.authenticate(SASLAuthentication.java: 247)
at org.jivesoftware.smack.tcp.XMPPTCPConnection.login(XMPPTCPConnection.java:374)
…
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.ja va:932)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.ja va:1)
at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)
at org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183)
at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:317)
at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:261)
at com.materna.mgmt.tree.xmpp.skeleton.impl.activator.Activator.start(Activator.ja va:33)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(BundleContextI mpl.java:711)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(Bundl eContextImpl.java:702)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextI mpl.java:683)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java :381)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.j ava:390)
at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java: 1176)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartL evelManager.java:559)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartL evelManager.java:544)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelMa nager.java:457)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(Star tLevelManager.java:243)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartL evelManager.java:438)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartL evelManager.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.jav a:230)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.j ava:340)
Sep 18, 2014 2:55:06 PM org.jivesoftware.smack.AbstractXMPPConnection callConnectionClosedOnErrorListener
Warnung: Connection closed with error
java.lang.NullPointerException
at org.jivesoftware.smack.util.stringencoder.Base64.decode(Base64.java:87)
at org.jivesoftware.smack.sasl.SASLMechanism.challengeReceived(SASLMechanism.java: 206)
at org.jivesoftware.smack.SASLAuthentication.challengeReceived(SASLAuthentication. java:325)
at org.jivesoftware.smack.SASLAuthentication.challengeReceived(SASLAuthentication. java:310)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPC onnection.java:1105)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$200(XMPPTCPCon nection.java:969)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnecti on.java:988)
java.lang.NullPointerException
at org.jivesoftware.smack.util.stringencoder.Base64.decode(Base64.java:87)
That seems to be the underlying cause. Will have a look.
Yes… The base64encoder field in the class org.jivesoftware.smack.util.stringencoder.Base64 is null!
Allright, the problem was that smack-java7 had no OSGi declarative service for its initializer, which cause the Base64 decoder not to be configured. This should be fixed with Add smack-java7 declarative service · 991d39a · Flowdalic/Smack · GitHub
I’ve also just uploaded a new alpha2-SNAPSHOT containing the changes.
@Jens: As always, could you try and report back? It’s my understanding that the initialization order problematic is still not solved, correct?
How did you declare the Smack dependency? It should be sufficient to just depend on smack-java7, as all other required artifacts are then pulled in automatically.
Best
Florian
I was not aware that I can use the “smack-java7” artifact… I have added the Snapshot repository and got the 4.0.1-alpha2-SNAPSHOT. There are still issues in the “smack-java7” bundle:
- MANIFEST.MF: org.jivesoftware.smackx/smack-java7-components.xml -> org.jivesoftware.smack/smack-java7-components.xml
- Import-Package: sun.security.util. Remove it from the header, but we should get rid of native sun packages. This becomes otherwise a big issue even in non-OSGi environments
But we have a fundamental startup and dependency problem. You can find a clean description here: What is the natural start order for package-dependent OSGI bundles (under Karaf)? - Stack Overflow. It means that the current startup approach via SCR is not working properly. I think my use case has just covered the problem. We must find a way that people using Smack in OSGi can rely that it is full functional. The current approach relies only on the OSGi package layer and that is the source of all the problems. I only see the following opportunities:
- Creation of an OSGi artifact e.g. “smack-osgi” where we merge the most critical bundles “smack-java7”, “smack-core”, “smack-extensions”, “smack-tcp”, “smack-resolver-javax”, “smack-sasl-javax” together. If necessary we can add OSGi specific classes here e.g. in a package like “org.jivesoftware.smack.osgi”. If “smack-extensions” should be optional than it must be a fragment again. There is no other safe way.
- Introduction of a service interaction layer where we can define dependencies between services (objects) explicitly.
- Mixture of both and resorting to the fragment approach, if interaction with Smack should still be done on the package and not the service layer.
Any thoughts?
MANIFEST.MF: org.jivesoftware.smackx/smack-java7-components.xml → org.jivesoftware.smack/smack-java7-components.xml
fixed, and uploaded as snapshot.
Import-Package: sun.security.util. Remove it from the header, but we should get rid of native sun packages. This becomes otherwise a big issue even in non-OSGi environments
Is this causing problems when using OSGi or just a “OMG, they use unofficial API which nobody should do” warning? If it’s the latter, then to be honest, I don’t see the immediate need to change it. I’d rather use the sun native API instead of duplicating the code from there into Smack for now. The mid-term goal is to write a XMPPHostnameVerifier that also takes the service name as third argument. But such code is security critical, errors can be disastrous, and I wont let anything like the ServerTrustManager disaster happen again. So until there is a bullet-proof, 100% XMPPHostnameVerifier, Smack is going to use the same code that is used by many others programs from the native sun API.
Back to the OSGi startup dependency problem. Just that I understand the situation correctly: If you would perform ‘(new SASLJavaXSmackInitializer).initialize()’ somewhere in your code before you are using Smack, then everything would be fine, right?
Right now, I don’t understand what makes smack-sasl-javax different from every other class where we use OSGi declarative services (e.g. smack-experimental). Could you elaborate that?
Is this causing problems when using OSGi or just a “OMG, they use unofficial API which nobody should do” warning? If it’s the latter, then to be honest, I don’t see the immediate need to change it. I’d rather use the sun native API instead of duplicating the code from there into Smack for now. The mid-term goal is to write a XMPPHostnameVerifier that also takes the service name as third argument. But such code is security critical, errors can be disastrous, and I wont let anything like the ServerTrustManager disaster happen again. So until there is a bullet-proof, 100% XMPPHostnameVerifier, Smack is going to use the same code that is used by many others programs from the native sun API.
The header entry MUST be removed because its a native package and not exported by the OSGi framework. The bundle cannot be resolved. It is up to the OSGi framework deployer to declare this package as beeing “bootdelegated”. This tells the classloader of the bundle to delegate a corresponding package request to the installed JRE. Only remove this entry and leave the “magic” of resolution to OSGi .
Back to the OSGi startup dependency problem. Just that I understand the situation correctly: If you would perform ‘(new SASLJavaXSmackInitializer).initialize()’ somewhere in your code before you are using Smack, then everything would be fine, right?
Right now, I don’t understand what makes smack-sasl-javax different from every other class where we use OSGi declarative services (e.g. smack-experimental). Could you elaborate that?
There is no real difference and the problem occurs also in all other bundles (smack-experimental, smack-extensions,…). I told you that I am using only a subset of Smack (smack-core, smack-extensions) and I am registering my own ExtensionProviders based on JAXB in Smack and do not care about those added by “smack-extensions” so the problem is covered, but still present.
Back to the OSGi startup dependency problem. Just that I understand the situation correctly: If you would perform ‘(new SASLJavaXSmackInitializer).initialize()’ somewhere in your code before you are using Smack, then everything would be fine, right?
That would be possible, but OSGi is a shared environment and the XMPP lib gets shared between multiple consumers. The call SASLJavaXSmackInitializer should be done only once and not by everyone who uses Smack in a bundle.
Only remove this entry and leave the “magic” of resolution to OSGi .
Just to be absolutely clear: You want me to remove sun.security.util from Import-Package in MANIFEST.MF of smack-java7. Right now, I don’t have an idea on how to do so, but I’m sure gradle will provide a way.
The call SASLJavaXSmackInitializer should be done only once and not by everyone who uses Smack in a bundle.
I was more thinking about placing the call in a static code block within the class using XMPPConnection, to mimic the initialization behavior of smack-core. The static code block should care of the code only being execute once even in an OSGi environment with multiple consumers (correct me if I’m wrong).
But let’s get one step back. There is smack-core, with looks for known Initializer classes (smack-extensions, smack-experimental, …) and loads them if they are available. On OSGi smack-core will never see any of the Initializer classes, because it does not declare a dependency on them. Which is what we want, because we don’t want hard dependencies from smack-core to any other smack component. All those dependencies are strictly optional.
That’s why we introduced OSGi declarative services for those components. It is my understanding that, if you pull in such a Smack component with OSGi, it’s service is activated, the Initializer is run, ie. performing the task that could not be accomplished by smack-core. Everything is fine, no dragons, no witches.
So even if smack-sasl-javax is activated after smack-core, everything should work (as long as you don’t perform any Smack/XMPP activity between the initialization of smack-core and e.g. smack-sasl-javax)
Help me here: Where is the issue? What am I missing?
Just to be absolutely clear: You want me to remove sun.security.util from Import-Package in MANIFEST.MF of smack-java7. Right now, I don’t have an idea on how to do so, but I’m sure gradle will provide a way.
You are right. The package must be removed from the Import-Package entry. In most cases it is enough to declare the “Export-Package” field explicitly.
But let’s get one step back. There is smack-core, with looks for known Initializer classes (smack-extensions, smack-experimental, …) and loads them if they are available. On OSGi smack-core will never see any of the Initializer classes, because it does not declare a dependency on them. Which is what we want, because we don’t want hard dependencies from smack-core to any other smack component. All those dependencies are strictly optional.
That’s why we introduced OSGi declarative services for those components. It is my understanding that, if you pull in such a Smack component with OSGi, it’s service is activated, the Initializer is run, ie. performing the task that could not be accomplished by smack-core. Everything is fine, no dragons, no witches.
So even if smack-sasl-javax is activated after smack-core, everything should work (as long as you don’t perform any Smack/XMPP activity between the initialization of smack-core and e.g. smack-sasl-javax)
Help me here: Where is the issue? What am I missing?
You are completely right. The problem resides on the consumer side - guys like me . Everyone who wants to use Smack from his bundle, imports the required Smack packages and starts working. You have stated the exact problem:
… (as long as you don’t perform any Smack/XMPP activity between the initialization of smack-core and e.g. smack-sasl-javax)
During the startup phase of the OSGi framework, all bundles are resolved and activated if resolvable on package layer. At the moment there is no gurantee that the framework activates the consumer’s bundle after activating smack-extensions or smack-sasl-javax. The activation order can be as follows: smack-core → consumer (exception, no SASLMechanisms) → smack-sasl-javax. There is a big difference between resolution (checking all package dependencies) and activation (causing SCR to do its job). The same applies to smack-experiments, …
This is our problem (extracted from What is the natural start order for package-dependent OSGI bundles (under Karaf)? - Stack Overflow):
In OSGi the bundle lifecycle is installed → resolved → starting → started. Import-Package and Export-Package only influence when the bundle goes from installed to resolved. So the framework makes sure all bundles you import packages from are resolved before your bundle but then your bundle only goes to the resolved state. Then in a second step the activators are called. So you can not assume the activators are called in the same order. If you need some initializations before […] then you should use OSGi services.
SCR replaces the old-fashioned Activator classes.
I don’t have an idea how that would look in detail, but
Creation of an OSGi artifact e.g. “smack-osgi” where we merge the most critical bundles
Could you elaborate that? It sounds like we would duplicate code here.
Introduction of a service interaction layer where we can define dependencies between services (objects) explicitly.
That sounds like the “best” solution. How would such a service interaction layer look like?
Mixture of both and resorting to the fragment approach, if interaction with Smack should still be done on the package and not the service layer.
We already had a fragments based approach in Smack and switched away from it. Resorting again to fragments seems like a step backwards.
Would be great if we could discuss this in a real-time chat, e.g. if you could join Smack (freenode) or open_chat@conference.igniterealtime.org (MUC).
I’m completely depending on your judgement here, since I’m not as nearly as far into OSGi as you. But there are already further consumers for Smack’s OSGi support (e.g. Bug 438006 – Update to Smack 4), so it’s definitely a often requested feature for which I’m willing to put some effort into a clean, nice and non-hacky solution, while not breaking Smack’s modularity or Android compatibility.
You are right. The package must be removed from the Import-Package entry. In most cases it is enough to declare the “Export-Package” field explicitly.
That seems like the an approach that will break as soon as you add further packages to the subproject. I went with
instruction ‘Import-Package’, ‘!sun.security.*, *’
instead.
Creation of an OSGi artifact e.g. “smack-osgi” where we merge the most critical bundles
Could you elaborate that? It sounds like we would duplicate code here.
On the other hand, i thought about creating smack-java7-osgi too, but there are some unanswered questions:
-
Should it depend on the minimal required set of Smack, or should it also depend on smack-extensions, smack-experimental
-
How would it guarantee the initialization order? E.g. would it provide something like SmackOsgi.init() which would initialize all of Smack.
How would this approach be compared to your “service interaction layer” idea?
Yes, that’s a much better solution!
It took me nearly the whole weekend, but I think I have some ideas now.
The main problem is, that there is no explicit distinction between which classes belong to Smack’s API, are for internal use or are part of the Smack’s SPI. A proper OSGi integration requires that interactions with services are based on Java interfaces, so there is always an internal service implementation and a corresponding (exported) interface. For the definition of a service layer, we must identify required services and must equip them with corresponding interfaces. It is no problem that API packages contain also some “real” implementation classes, but it is always good practice to avoid this whenever possible. Have a look here: Eclipse/API Central - Eclipsepedia , especially Naming Conventions - Eclipsepedia and SPI description - Eclipsepedia. The refactoring would help to pass the integration in ECF, but this is a hard and painful job because it means tearing a hole in already working structures and communication flows. What we need is a package structure like this:
• org.org.jivesoftware.smack.[.*]: API-Packages (mostly Java interfaces, used by consumers and exported in OSGi)
• org.org.jivesoftware.smack.impl.[.*]: Internal Packages (never used by consumers which belong to the implementation classes, in most cases not exported in OSGi, but used and referenced in Smack standalone. That’s the reason why I don’t like naming these packages “internal” as recommended by Eclipse)
• org.org.jivesoftware.smack.spi.[.*]: Service Provider Interface for things offered by other providers and which are required for Smack to do its job (e.g. SASLMechanism, Base64Encoder, …). These classes are never used by “normal” Smack users, if so they must belong to the API.
The question is:
Is this the way it goes… I would say: Yes. It will help you to clearly separate API, Implementation and SPI. All consumers and developers would benefit from this step because usage becomes much easier and structures become more maintainable.
Without properly defined interface-based API packages, I see no changes for getting a service layer running in OSGi otherwise we must resort to OSGi’s package layer and are confronted with the problems we currently have (start-oder, fragments, …).
On the other hand, i thought about creating smack-java7-osgi too, but there are some unanswered questions:
- Should it depend on the minimal required set of Smack, or should it also depend on smack-extensions, smack-experimental
- How would it guarantee the initialization order? E.g. would it provide something like SmackOsgi.init() which would initialize all of Smack.
How would this approach be compared to your “service interaction layer” idea?
You asked for code redundancy… Agreed, that is a nightware, but what I meant was only to create a specific OSGi assembly defined in Gradle in the form of “put smack-core, smack-tcp, …” inside a jar called smack-java7-osgi. That would avoid all of our starting issues right now, but this solution would also stop on OSGi’s package layer. There are no services available to interact with. The advantage is that code once written by consumers runs with Smack standalone and out of the box in an OSGi environment without any need for modifications. But this approach requires a monolytic bundle containing all of Smack extensions otherwise we are still in trouble with the starting order. Not very nice…
I just checked the Smack’s current state… I am pretty unsure what is the right approach, maybe for the first version we can try to wrap things and live with the fact that most of things are implementation classes and no interfaces. But we could add a service layer (based on proper Java interfaces) in the form of e.g.
- XmppTcpConnectionBuilderFactory (#newBuilder) -> XmppTcpConnectionBuilder (#build) -> XmppTcpConnection
- ChatBuilderFactory (#newBuilder(XmppConnection) -> ChatBuilder (#build) -> Chat
- MessageBuilderFactory(#newBuilder) -> MessageBuilder (#build) -> Message
- PubSubManagerBuilderFactory(#newBuilder) -> PubSubManagerBuilder (#build) -> PubSubManager
I would be cool, if those things could be reusable also in non-OSGi environments.
Some OSGi services would be:
- ExtensionManager (offered by smack-core)
- ConnectionManager (offered by smack-core)
- XmppTcpConnectionBuilderFactory (offered by smack-tcp, depends on ConnectionManager)
- PubSubManagerBuilderFactory (offered by smack-extensions, depends on ExtensionManager)
- SaslProvider (offered by smack-sasl, required by smack-core)
- PlatformToolkit (offered by smack-android or smack-java7, required by smack-core)
Thanks for your time you have but into thinkering and researching this. This appears to be something feasible only as long-term solution (maybe mid-term), since it would involve a lot of API changes (and code changes). But I think it’s definitely the way Smack should pursue. But right know, my primary goal is to get the 4.1 release out sometime in Q4 2014.
So as short term solution, I’m thinking about something like smack-java7-osgi, which depends on smack-extensions, smack-experimental and smack-java7 and provides a method SmackOsgi.activate(), that will have Smack’s OSGi bundles in active state when it returns. As far as I understand OSGi, this should be accomplished by calling the activate() methods of the respective SmackInitalizers of the Smack components. I’m not sure, but it appears to be that this would make Smack’s declarative services obsolete (correct me if I’m wrong).
Some remarks:
I would be cool, if those things could be reusable also in non-OSGi environments.
Ack
- SaslProvider (offered by smack-sasl, required by smack-core)
ATM the is no subproject smack-sasl, only smack-sasl-provided and smack-sasl-javax with the implementations for the platforms. Does this mean that there should be a smack-sasl subproject? Or shouldn’t SaslProvider go into smack-core?