Introducing Hazelcast ... a new way to cluster Openfire!

Garan_Sink · November 22, 2012, 4:15pm

Thanks for the direction, I am using the method mentioned above to specify nodes instead of multicast.

I’m getting this in the logs:

2012.11.22 11:03:37 org.jivesoftware.openfire.plugin.BroadcastPlugin - Hazelcast Instance is not active!

java.lang.IllegalStateException: Hazelcast Instance is not active!

at com.hazelcast.impl.FactoryImpl.initialChecks(FactoryImpl.java:711)

at com.hazelcast.impl.MProxyImpl.beforeCall(MProxyImpl.java:102)

at com.hazelcast.impl.MProxyImpl.get(MProxyImpl.java:112)

at com.jivesoftware.util.cache.ClusteredCache.get(ClusteredCache.java:86)

at org.jivesoftware.util.cache.CacheWrapper.get(CacheWrapper.java:121)

at org.jivesoftware.openfire.spi.RoutingTableImpl.removeComponentRoute(RoutingTabl eImpl.java:821)

at org.jivesoftware.openfire.component.InternalComponentManager.removeComponent(In ternalComponentManager.java:238)

at org.jivesoftware.openfire.component.InternalComponentManager.removeComponent(In ternalComponentManager.java:211)

at org.jivesoftware.openfire.plugin.BroadcastPlugin.destroyPlugin(BroadcastPlugin. java:120)

at org.jivesoftware.openfire.container.PluginManager.shutdown(PluginManager.java:1 46)

at org.jivesoftware.openfire.XMPPServer.shutdownServer(XMPPServer.java:948)

at org.jivesoftware.openfire.XMPPServer.access$700(XMPPServer.java:145)

at org.jivesoftware.openfire.XMPPServer$ShutdownHookThread.run(XMPPServer.java:896 )

Garan_Sink · November 26, 2012, 3:06pm

An update to my issue above… I removed all other plugins and this error went away however I was still getting another error, which turned out to simply be a configuration file mis-type!

Now both instances show up as they should. On to testing, awesome!

olderboy1 · January 22, 2013, 3:17am

Hello

My Openfire Version 3.7.1. I fonund this package for 3.7.1. I installed it.

It’s OK. I can login to server.

I tested MUC with two openfire server and cluster package. It’s OK. Can work. no Problems.

But If I added to three server. I get the error message. It’s not work. Server always tell me “No response from server”

It’s my error log.

org.jivesoftware.openfire.IQRouter - Could not route packet

java.lang.ClassCastException: com.hazelcast.impl.MemberImpl cannot be cast to java.lang.Comparable

at java.util.TreeMap.put(Unknown Source)

at java.util.TreeSet.add(Unknown Source)

at com.jivesoftware.util.cache.CoherenceClusteredCacheFactory.doClusterTask(Cohere nceClusteredCacheFactory.java:221)

at org.jivesoftware.util.cache.CacheFactory.doClusterTask(CacheFactory.java:499)

at org.jivesoftware.openfire.component.InternalComponentManager.process(InternalCo mponentManager.java:512)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToComponent(RoutingTableImp l.java:352)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:237)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:324)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:121)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)

at org.jivesoftware.openfire.component.InternalComponentManager.sendPacket(Interna lComponentManager.java:281)

at org.jivesoftware.openfire.plugin.SearchPlugin.processPacket(SearchPlugin.java:2 43)

at org.jivesoftware.openfire.component.InternalComponentManager.checkDiscoSupport( InternalComponentManager.java:458)

at org.jivesoftware.openfire.component.InternalComponentManager.addComponent(Inter nalComponentManager.java:171)

at org.jivesoftware.openfire.plugin.SearchPlugin.initializePlugin(SearchPlugin.jav a:164)

at org.jivesoftware.openfire.container.PluginManager.loadPlugin(PluginManager.java :483)

at org.jivesoftware.openfire.container.PluginManager.access$300(PluginManager.java :80)

at org.jivesoftware.openfire.container.PluginManager$PluginMonitor.run(PluginManag er.java:1067)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)

at java.util.concurrent.FutureTask.runAndReset(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101 (Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodi c(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknow n Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

How to slove this problem??

Thanks for your help.

Tom_Evans1 · January 22, 2013, 3:59am

This issue has been fixed in the latest nightly builds for 3.7.2. Unfortuately, the fix was applied after the custom plugin build (back-ported for 3.7.1) was created. There are three possible solutions:

Update to the latest nightly build for 3.7.2 including this fix (among others)
Limit your cluster size to two nodes with the existing 3.7.1 build
Apply the simple fix to the 3.7.1 code base and rebuild (here’s a link)

Hope that helps!

olderboy1 · January 22, 2013, 8:42am

Hello

Thanks for your reply.

I try re-compile openfire 3.7.1 with clustering.

But I got this error.

I’m not R&D. I don’t know why it. Can you give me a suggestion?? Thanks a lot.

[javac] Compiling 27 source files to /opt/openfire_src/work/plugins-dev/hazelcast/target/classes

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/openfire/sess ion/RemoteSession.java:172: cannot find symbol

[javac] symbol : method getClusterNodeInfo(byte[])

[javac] location: class org.jivesoftware.util.cache.CacheFactory

[javac] ClusterNodeInfo info = CacheFactory.getClusterNodeInfo(nodeID);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/openfire/sess ion/RemoteSession.java:189: cannot find symbol

[javac] symbol : method getClusterNodeInfo(byte[])

[javac] location: class org.jivesoftware.util.cache.CacheFactory

[javac] ClusterNodeInfo info = CacheFactory.getClusterNodeInfo(nodeID);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Ca cheListener.java:67: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] NodeID nodeID = NodeID.getInstance(StringUtils.getBytes(event.getMember().getUuid()));

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:417: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:442: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:475: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:613: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] nodePresences.put(NodeID.getInstance(StringUtils.getBytes(event.getMember().get Uuid())),

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:616: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] ClusterManager.fireJoinedCluster(StringUtils.getBytes(event.getMember().getUuid ()), true);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterListener.java:623: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] byte[] nodeID = StringUtils.getBytes(event.getMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usterExternalizableUtil.java:43: com.jivesoftware.util.cache.ClusterExternalizableUtil is not abstract and does not override abstract method readSerializableMap(java.io.DataInput,java.util.Map<java.lang.String,? extends java.io.Serializable>,java.lang.ClassLoader) in org.jivesoftware.util.cache.ExternalizableUtilStrategy

[javac] public class ClusterExternalizableUtil implements ExternalizableUtilStrategy {

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:229: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] return StringUtils.getBytes(oldest.getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:238: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] return StringUtils.getBytes(cluster.getLocalMember().getUuid());

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:253: members is already defined in doClusterTask(org.jivesoftware.util.cache.ClusterTask)

[javac] Set members = new HashSet();

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:286: cannot find symbol

[javac] symbol : method getString(byte[])

[javac] location: class org.jivesoftware.util.StringUtils

[javac] logger.warn(“Requested node " + StringUtils.getString(nodeID) + " not found in cluster”);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:351: cannot find symbol

[javac] symbol : method getString(byte[])

[javac] location: class org.jivesoftware.util.StringUtils

[javac] logger.warn(“Requested node " + StringUtils.getString(nodeID) + " not found in cluster”);

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cache/Cl usteredCacheFactory.java:369: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] if (Arrays.equals(StringUtils.getBytes(member.getUuid()), nodeID)) {

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cluster/ HazelcastClusterNodeInfo.java:48: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] nodeID = NodeID.getInstance(StringUtils.getBytes(member.getUuid()));

[javac] ^

[javac] /opt/openfire_src/src/plugins/hazelcast/src/java/com/jivesoftware/util/cluster/ HazelcastClusterNodeInfo.java:50: cannot find symbol

[javac] symbol : method getBytes(java.lang.String)

[javac] location: class org.jivesoftware.util.StringUtils

[javac] seniorMember = ClusterManager.getSeniorClusterMember().equals(StringUtils.getBytes(member.getU uid()));

[javac] ^

[javac] Note: Some input files use or override a deprecated API.

[javac] Note: Recompile with -Xlint:deprecation for details.

[javac] Note: Some input files use unchecked or unsafe operations.

[javac] Note: Recompile with -Xlint:unchecked for details.

[javac] 18 errors

[trycatch] Caught exception: Compile failed; see the compiler error output for details.

[echo] Error building plugin: hazelcast. Exception:

[echo] /opt/openfire_src/build/build.xml:1310: Compile failed; see the compiler error output for details.

Nick_Cappelletti · February 12, 2013, 6:41pm

Like others, I’m having similar issues with the tcp-ip settings in hazelcast working properly. I have added both of my openfire servers to the config as so:

jabbertest.switchtower.com:5701

jabbertest2.switchtower.com:5701

I’ve verified the server is listening on the 5701 port:

netstat -lan | grep 5701

tcp 0 0 0.0.0.0:5701 0.0.0.0:* LISTEN

Doing a tcpdump between the servers, I see no communication with the servers on that port. There are no iptables rules and the HW firewalls these servers sit behind are configured to allow full IP communciation.

Tom_Evans1 · February 12, 2013, 6:57pm

Check the nohup.out file in the log directory (console output) to determine whether your Openfire installation is finding your customized Hazelcast configuration file. You should also be able to see in this log file exactly which servers have joined the cluster.

Youngho_Kim · April 21, 2013, 12:00pm

Hi @Tom Evans.

Good to see you!

I has composing Openfire HA cluster on Amazon AWS using hazelcast cluster plugin.

Unfortunately! Amazon Service do not support direct connection between their regions.

So, I use the hazelcast cluster plugin on openfire. It seemed to be like which did not have a problem.

But, Increasing of hazelcast cluster node has occured big problem.

First, this service is very unstable between cluster node.

Second, this service has occured latency between clients message about 2~3 second. It tested just two clients.

So, I was find replication property on hazelcast document about “wan-replication”. and I was configured your hazelcast configuration xml file.

But, this do not operate. What is problem on my configuration?

like this:

com.hazelcast.impl.wan.WanNoDelayReplication</replication-imp l>

10.0.0.50:5701

com.hazelcast.impl.wan.WanNoDelayReplication</replication-imp l>

10.1.0.50:5701

kentzen · August 15, 2013, 7:36am

The issuses of after deployed cluster with Hazelcast Clustering Plugin of openfire .

I have three server nodes : A (10.0.1.113),B (10.0.1.176),C (10.0.1.158).

And all of them share a same Mysql database. Initially, I installed Openfire to B & C server.After that I installed Openfire to A. Such an installation made domain of all servers same as domain of A server.All server domains are 10.0.1.113 now. I used Strophe.js accessing A node successfully via BOSH After deployed cluster with Hazelcast Clustering Plugin. But might be owing to domains of B and C node are same as A’s , lead to I can not access B and C via BOSH with Strophe.js but the clients software like Spark can access all of them and works fine. Accessing B and C via BOSH made errors below:

Now my issue is that I need to access the cluster by using BOSH.Is it enough access only A node or I need to be able to access all nodes via BOSH? If I wanna access all nodes via BOSH in the meantime,how should I deploy cluster of openfire.

Here is configuration of Hazelcast

224.2.2.3

54327

10.0.1.113

10.0.1.176

10.0.1.158

10.0.1.113

I appreciate any help ! I hope you could forgive my awful english and understand my issue description.

Dele_Olajide · August 15, 2013, 9:14am

What is your domain name? Have you created a DNS record for your domain name pointing to all three IP addresses?

kentzen · August 15, 2013, 10:07am

No,I have not created.I merely use an IP among these three nodes such as 10.0.1.113.And I test the cluster in Local Area Nework.

Thanks for reply!

Dele_Olajide · August 15, 2013, 2:24pm

I don’t think it will work. The domain name should be a valid DNS name that resolves via the network to the next IP address by cycling through the 3 IP addresses

Alex_Mateescu · August 15, 2013, 4:56pm

I think having A records (IPs) in DNS foe each server is necessary as a fallback mechanism. It’s possible (and recommended) to have SRV records instead.

kentzen, maybe this will help you: http://wiki.xmpp.org/web/SRV_Records

kentzen · August 16, 2013, 1:02am

Yeah.Dele and Alex,Thanks you a lot .It works now.

Here is my steps:

My server is Redhat6 32 bit.

1.Modify all nodes /etc/hosts file .Plus code as following:

127.0.0.1 openfire #Any valid DNS name you want.

2.Change ‘setup’ option to false within ‘/openfire/conf/openfire.xml’,this step means that you could reset configuration for openfire.And type a DNS name you just bound with IP ‘127.0.0.1’ into ‘domain’ option.

3.Check ‘xmpp.domain’ field in ‘ofProperty’ table of openfire database,see if its value was set to ‘admin@openfire’ (or admin@).Change it if it isn’t.

4.Restart openfire.

(note : Do these steps to all nodes,and domain name of each node is the same like ‘openfire’ )

Via steps above .It should work fine! Please contact me if you confuse!

Alex_Mateescu · August 16, 2013, 7:22am

I doubt hardcoding IPs left and right will get you too far, but if it works for now… enjoy.

kentzen · August 16, 2013, 7:34am

Lol .Thanks.

kentzen · August 20, 2013, 3:12am

Which is better , Openfire cluster + nginx or Openfire cluster + Connection Manager?

Udhayakumar · March 25, 2015, 3:13pm

Am able to create openfire cluster with 3 node using hazelcast plugin. I have tried with TCP and Multicast both. But problem is its automatically discovering which i dont want to be part of my cluster. How can we restrict other openfire node from this cluster environment?

Please help. Thanks in advance.

Tom_Evans1 · March 26, 2015, 5:30pm

You can change the multicast group and/or port number in the classes/hazelcast-cache-config.xml file to create individual clusters:

224.2.2.3

54327

Udhayakumar · March 27, 2015, 6:49am

Thanks Tom.

It means, i can have my own multicast configuration. For example,

           <multicast-group>420.3.1.2</multicast-group>*

           <multicast-port>42031</multicast-port>*

*

Is it correct?

and Am deploying my server in EC2 i don’t want to open to many ports, how shall i restrict my servers to look in to only one port, i tried tcp/ip config as well but that doesn’t work. Am trying with private ip.

172.31.34.200:5701*
172.31.34.233:5701*
*

Please help me to configure cluster in secured manner?