Status of Clustering?

I’ve been running Openfire 3.6.4 for quite some time, and we recently have a need to do some clustering.

I’d apperciate some confimation or corrections on what I’ve spent the last 12 hours learning:

  • Coherence 3.3.1 is no longer availible (Oracle has 3.6, 3.5, and 3.4 on their website)
  • Clustering plugin in 3.6.4 requires Coherence 3.3.1 (Because it doesn’t work with 3.4 or higher)
  • Clustering plugin in 3.7.0 is the same plugin as 3.6.4
    Mike

I’m afraid there is noone here to fully answer questions about Openfire-Coherence combo. Jive Software has opensourced their clustering plugin, but noone is developing it further (a few volunteers are doing work on JBoss, if still working, http://community.igniterealtime.org/thread/38847). So i think plugin is still the same and if it doesn’t work with 3.4 then there is no other options. You can try emailing gato at jivesoftware.org He’s the original author of Openfire clustering plugin.

Thanks for the info. It appears the JBoss plugin development has stalled.

I’ve pretty much given up on clustering, I was just trying to collect / confirm the facts that I’ve run across. I don’t think I’m going to bother Gato, as it seems impossible to obtain 3.3.1 unless someone emails it to me.

The changes to the clustering plugin to use the latest 3.7 coherence affect the files:

  1. ClusteredCache.java
  2. CoherenceClusteredCacheFactory.java
  3. CoherenceExternalizableUtil.java
  4. CoherenceInfo.java
  5. system-clustering-node.jsp

I can supply the diffs against a 3.70. OpenFire, for a 3.7.0 Coherence, but I expect that it would work for a 3.7.1 Coherence as well. I expect netiquette suggests I should not put them in the forum

Why can’t you put the in the forum? We would be glad to get a patch. Btw, is the latest Coherence version also numbered 3.7.0? That’s confusing

This patch file was generated by NetBeans IDE

Following Index: paths are relative to: openfire_3_7_0/src/plugins/clustering/src

This patch can be applied using context Tools: Patch action on respective folder.

It uses platform neutral UTF-8 encoding and \n newlines.

Above lines and this line are ignored by the patching process.

Index: java/com/jivesoftware/util/cache/ClusteredCache.java

— java/com/jivesoftware/util/cache/ClusteredCache.java Base (BASE)

+++ java/com/jivesoftware/util/cache/ClusteredCache.java Locally Modified (Based On LOCAL)

@@ -190,7 +190,7 @@

public long getCacheHits() {

if (map instanceof NearCache) {

  •         return ((NearCache)map).getCacheHits();
    
  •         return ((NearCache)map).getCacheStatistics().getCacheHits();
    

}

else if (backingCache != null) {

return backingCache.getCacheHits();

@@ -202,7 +202,7 @@

public long getCacheMisses() {

if (map instanceof NearCache) {

  •         return ((NearCache)map).getCacheMisses();
    
  •         return ((NearCache)map).getCacheStatistics().getCacheMisses();
    

}

else if (backingCache != null) {

return backingCache.getCacheMisses();

Index: java/com/jivesoftware/util/cache/CoherenceClusteredCacheFactory.java

— java/com/jivesoftware/util/cache/CoherenceClusteredCacheFactory.java Base (BASE)

+++ java/com/jivesoftware/util/cache/CoherenceClusteredCacheFactory.java Locally Modified (Based On LOCAL)

@@ -93,7 +93,7 @@

}

else {

com.tangosol.net.CacheFactory.getCache(“opt-$cacheStats”);

  •                taskService = com.tangosol.net.CacheFactory.getInvocationService("OpenFire Cluster Service");
    
  •                taskService = (InvocationService) com.tangosol.net.CacheFactory.getService("OpenFire Cluster Service");
    

// Update the running state of the cluster

state = cluster != null ? State.started : State.stopped;

Index: java/com/jivesoftware/util/cache/CoherenceExternalizableUtil.java

— java/com/jivesoftware/util/cache/CoherenceExternalizableUtil.java Base (BASE)

+++ java/com/jivesoftware/util/cache/CoherenceExternalizableUtil.java Locally Modified (Based On LOCAL)

@@ -286,7 +286,7 @@

}

public Serializable readSerializable(DataInput in) throws IOException {

  •    return ExternalizableHelper.readSerializable(in);
    
  •    return (Serializable) ExternalizableHelper.readSerializable(in);
    

}

public void writeSafeUTF(DataOutput out, String value) throws IOException {

Index: java/com/jivesoftware/util/cluster/CoherenceInfo.java

— java/com/jivesoftware/util/cluster/CoherenceInfo.java Base (BASE)

+++ java/com/jivesoftware/util/cluster/CoherenceInfo.java Locally Modified (Based On LOCAL)

@@ -64,7 +64,7 @@

*/

public static Map getNodeInfo() {

  •    InvocationService service = com.tangosol.net.CacheFactory.getInvocationService("OpenFire Cluster Service");
    
  •    InvocationService service = (InvocationService) com.tangosol.net.CacheFactory.getService("OpenFire Cluster Service");
    

// Run cluster-wide stats query

Map results = service.query(new AbstractInvocable() {

@@ -107,7 +107,7 @@

*/

public static void clearCacheStats() {

  •    InvocationService service = com.tangosol.net.CacheFactory.getInvocationService("OpenFire Cluster Service");
    
  •    InvocationService service = (InvocationService) com.tangosol.net.CacheFactory.getService("OpenFire Cluster Service");
    

service.execute(new AbstractInvocable() {

public void run() {

Index: web/system-clustering-node.jsp

— web/system-clustering-node.jsp Base (BASE)

+++ web/system-clustering-node.jsp Locally Modified (Based On LOCAL)

@@ -123,7 +123,7 @@

}

// Get the cache stats object:

  • Map cacheStats = com.tangosol.net.CacheFactory.getReplicatedCache(
  • Map cacheStats = com.tangosol.net.CacheFactory.getCache(

“opt-$cacheStats”, com.tangosol.net.CacheFactory.class.getClassLoader());

// Decimal formatter for nubmers

@@ -354,8 +354,8 @@

double memUsed = (double) size / (1024 * 1024);

double totalMem = (double) maxSize / (1024 * 1024);

  •    double freeMem = 100 - 100 * memUsed / totalMem;
    
  •    double usedMem = 100 * memUsed / totalMem;
    

\ No newline at end of file

  •    double freeMem = 100 - 100 * memUsed / Math.max(1, totalMem);
    
  •    double usedMem = 100 * memUsed / Math.max(1, totalMem);
    

\ No newline at end of file

long hits = theStats[3];

long misses = theStats[4];

double hitPercent = 0.0;

Here is the diff on the 5 files. I would suggest checking it out and validating that everything still matches up, and I didn’t forget anything.

Oh, could you please attach it as a diff file. I don’t want to spoil anything while copying from the web (you know, various hidden chars).

I just tried a copy and paste from the forum, and it looked pretty good.

I will attach here.
openfire.diff.zip (1404 Bytes)

The latest coherence appears to actually be 3.7.1, but I haven’t used it yet, just the 3.7.0. In another thread there was a question about coherence license requirements. Openfire requires Enterprise Edition. I put a longer response in there, with the way to validate that by putting a license-config section into the tangosol-coherence-override.xml, like below.

EE

prod

If you use SE, rather than EE, the error.log will confirm that Openfire needs an Enterprise license.

I have attached your patch to OF-491. But i have a comment: I wonder how this should be done. I don’t know exactly, but maybe Openfire 3.7.x still works with Coherence 3.3.x and if we incorporate this patch, maybe it will brake such compatibility and Openfire will only work with 3.7 version of Coherence. But probably we are good to go, as ones who are using older Coherence probably are using older Openfire anyway.

I cannot recall that I validated the changes with a 3.3.1 Coherence, so it would be reasonable to proceed with caution. I would presume that anyone taking the patch would ensure that their system is working afterwards and that they have some form of testing in place, whether it was for a 3.3.1 or a 3.7 Coherence.

I was writing a response, but I realized I started this thread, so I don’t need to restate that 3.3.1 is not availible.

I think if you make a note on the release notes that it now requires Coherence 3.7.0 or higher, (Yes, 3.7.0 was the version) everything will be fine.

Now I’m off to find that other thread you mentioned.

BTW, what type of coherence is required, JAVA? C++ ? .NET?

You need Coherence for Java, since Openfire is a java application.

I took your patch and integrated it with the 1.2.1 beta clustering plugin. Works great with Coherence 3.7.1

Only issue I have noticed so far is that under cluster status the ‘packet size’ is 0 bytes for both nodes. Not sure if this is just aestetic or if it impacts the cluster.

I believe it was 1468 using the plugin based on coherence 3.3.1.

That element comes from “packet-publisher/packet-size/maximum-length” in the cluster config, as referenced from line 94 in system-clustering-node.jsp, as referenced at line: 467.

I could not find this element defined in the coherence-config.xml or tangosol-coherence-override.xml, and so the previous value would have been a default. It may be called something different now.

That line was not modified in the diff, so the difference might be that the element is a different name now.

Looking at the Coherence Developers Guid (A-48) suggests that you might find it as “packet-publish/maximum-length”.

I haven’t tried that yet.

Tried it as packet-publisher/maximum-length and packet-publisher/packet-size, both return 0.

It may be that 0 means a default, rather than an specific setting.

A-49 suggests that maximum-length is a sub-element of packet-publisher/packet-size, so it would be the full string.

Neither the Coherence tangosol-choherence-override.xml, or tangoxol-coherence.xml (in the jar) have the packet size, but both have a packet-publisher element. The tangosol override in OF does not have a packet-publisher, so I could see a default of 0 being returned.

Hi, I’ve got clustering (openfire 3.7.1 with the coherence jars from http://community.igniterealtime.org/message/213130#213130 ) to work for me but I’m experiencing problems with pep. my chat works fine.

however when clustering is on, I get this error on every pep publish attempts using strophe:

2011.12.19 09:28:12 org.jivesoftware.openfire.handler.IQHandler - Internal server error

(Wrapped) java.io.NotSerializableException: org.jivesoftware.openfire.commands.AdHocCommandManager at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)

at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)

at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)

at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)

at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)

at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)

at com.tangosol.util.ExternalizableHelper.writeSerializable(ExternalizableHelper.j ava:2120)

at com.tangosol.util.ExternalizableHelper.internalWriteObject(ExternalizableHelper .java:2501)

at com.tangosol.util.ExternalizableHelper.toBinary(ExternalizableHelper.java:169)

at com.tangosol.util.ExternalizableHelper$2.convert(ExternalizableHelper.java:3877 )

at com.tangosol.util.ConverterCollections$ConverterObservableMap.put(ConverterColl ections.java:1796)

at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Distributed Cache$ViewMap.put(DistributedCache.CDB:1)

at com.tangosol.coherence.component.util.SafeNamedCache.put(SafeNamedCache.CDB:1)

at com.tangosol.net.cache.CachingMap.put(CachingMap.java:929)

at com.tangosol.net.cache.CachingMap.put(CachingMap.java:888)

at com.tangosol.net.cache.CachingMap.put(CachingMap.java:800)

at com.jivesoftware.util.cache.ClusteredCache.put(ClusteredCache.java:143)

at org.jivesoftware.util.cache.CacheWrapper.put(CacheWrapper.java:129)

at org.jivesoftware.openfire.pep.PEPServiceManager.loadPEPServiceFromDB(PEPService Manager.java:148)

at org.jivesoftware.openfire.pep.PEPServiceManager.getPEPService(PEPServiceManager .java:78)

at org.jivesoftware.openfire.pep.IQPEPHandler.handleIQ(IQPEPHandler.java:337)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:65)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:372)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:121)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:337)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:93)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:302)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:194)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:169)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Thread.java:662)

Please advice,

Best Regards,

Stevenson Lee

It would appear that when the PEPServiceManager.load - line 148, is running that it will do a put to a clustered cached map, putting a newly minted PEPService. However, though PEPService is Serializable through its implementation of Cacheable, the AdHocCommandManager it contains at line 151 does not implement the Serializable interface.

Thus when then go to the cache, and they are serialized (or attempted to be) by Cohence, you get the NotSerializableException in AdHocCommandManager.

You will need to investigate a bit, since the Concurrent maps in AdHocCommandManager, which will need to be serialized, contain AdHocCommand instances, which do not implement Serializable (though its String conent would be inherently Serializable).