Hazelcast Clustering and 2 Nodes

Openfire Openfire Support
1

Hello everyone,

I am having a rough time trying to get this working right. Here is my setup:

IM01 (192.168.70.71 (Primary NIC) 192.168.70.73 (Microsoft NLB (Private)))

IM02 (192.168.70.72 (Primary NIC) 192.168.70.74 (Microsoft NLB (Private)))

NLB Cluster (192.168.70.75)

OpenFire 3.8.0

HazelCast 1.0.4 (Downloaded using the OpenFire Plugin Menu)

As noted above, I have two NICs per server. Both servers are virtual.

Both have Microsoft Network Load Balancing setup to share their Private NIC to create the NLB Cluster. Both NLBs are configured to use Multicasting as there are issues with Unicast and VMWare vitualization. I want the NLB cluster IP (192.168.70.75) to be the address that the end users connect to to connect to the IM Cluster. The obvious reason is to allow redundant IM servers with one address for the users to connect to. If a serer goes down, no big deal the IM server is still up.

With that being said, my current setup (hazelcast-cache-config.xml) is:

IM01:

    <port auto-increment="true">5701</port>

    <join>

        <multicast enabled="false"> <!-- Used for multicasting auto detection. -->

            <multicast-group>224.2.2.3</multicast-group>

            <multicast-port>54327</multicast-port>

        </multicast>

        <tcp-ip enabled="true"> <!-- Used for Direct TCP traffic to other nodes. -->

            <hostname>IM01</hostname>

            <hostname>IM02</hostname>

            <interface>192.168.70.73</interface>

            <interface>192.168.70.74</interface>

        </tcp-ip>

        <aws enabled="false"/> <!-- Used for EC2 Auto Discovery -->

    </join>

    <interfaces enabled="true">

        <interface>192.168.70.73</interface>

        <public-address>192.168.70.75</public-address>

    </interfaces>

IM02:

    <port auto-increment="true">5701</port>

    <join>

        <multicast enabled="false"> <!-- Used for multicasting auto detection. -->

            <multicast-group>224.2.2.3</multicast-group>

            <multicast-port>54327</multicast-port>

        </multicast>

        <tcp-ip enabled="true"> <!-- Used for Direct TCP traffic to other nodes. -->

            <hostname>IM01</hostname>

            <hostname>IM02</hostname>

            <interface>192.168.70.73</interface>

            <interface>192.168.70.74</interface>

        </tcp-ip>

        <aws enabled="false"/> <!-- Used for EC2 Auto Discovery -->

    </join>

    <interfaces enabled="true">

        <interface>192.168.70.74</interface>

        <public-address>192.168.70.75</public-address>

    </interfaces>

As you can see, I have both IM01 & IM02 pointed to their NLB interface and the public address is the NLB cluster address.

I get the following in the OpenFire window. Clustering isn’t working. When IM01 is online the user can connect to 192.168.70.75. Even if IM02 is online, the user can’t connect when IM01 is offline. The Clustering page on the OpenFire admin console only shows the respective server, not both servers.

Here is the OpenFire text (Note the red text. The blue text shows it has started.):

IM01 (When IM02 is offline)

Openfire 3.8.0 [Feb 22, 2013 9:04:00 AM]

Admin console listening at:

http://im:9090

https://im:9091

Starting Client Control Plugin

Starting Monitoring Plugin

Starting Hazelcast Clustering Plugin

Feb 22, 2013 9:04:06 AM com.hazelcast.config.ClasspathXmlConfig

INFO: Configuring Hazelcast from ‘hazelcast-cache-config.xml’.

Feb 22, 2013 9:04:07 AM com.hazelcast.impl.AddressPicker

INFO: Resolving domain name ‘IM01’ to address(es): [192.168.70.71, 192.168.70.73, 192.168.70.75]

Feb 22, 2013 9:04:12 AM com.hazelcast.impl.AddressPicker

INFO: Resolving domain name ‘IM02’ to address(es): [192.168.70.72]

Feb 22, 2013 9:04:12 AM com.hazelcast.impl.AddressPicker

INFO: Interfaces is enabled, trying to pick one address matching to one of: [IM01/192.168.70.73]

Feb 22, 2013 9:04:12 AM com.hazelcast.impl.AddressPicker

INFO: Prefer IPv4 stack is true.

Feb 22, 2013 9:04:13 AM com.hazelcast.impl.AddressPicker

INFO: Picked Address[IM01]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true

Feb 22, 2013 9:04:13 AM com.hazelcast.system

INFO: [IM01]:5701 [openfire] Hazelcast Community Edition 2.5 (20130115) starting at Address[IM01]:5701

Feb 22, 2013 9:04:13 AM com.hazelcast.system

INFO: [IM01]:5701 [openfire] Copyright © 2008-2012 Hazelcast.com

Feb 22, 2013 9:04:13 AM com.hazelcast.impl.LifecycleServiceImpl

INFO: [IM01]:5701 [openfire] Address[IM01]:5701 is STARTING

Feb 22, 2013 9:04:13 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM01]:5701 [openfire] Connecting to possible member: Address[192.168.70.74]:5702

Feb 22, 2013 9:04:13 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM01]:5701 [openfire] Connecting to possible member: Address[192.168.70.74]:5701

Feb 22, 2013 9:04:13 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM01]:5701 [openfire] Connecting to possible member: Address[192.168.70.73]:5703

Feb 22, 2013 9:04:13 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM01]:5701 [openfire] Connecting to possible member: Address[192.168.70.74]:5703

Feb 22, 2013 9:04:13 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM01]:5701 [openfire] Connecting to possible member: Address[192.168.70.73]:5702

Feb 22, 2013 9:04:13 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM01]:5701 [openfire] Connecting to possible member: Address[192.168.70.73]:5701

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 1961 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM01]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.73:1961

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 5701 accepted socket connection from /192.168.70.73:1961

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM01]:5701 [openfire] Wrong bind request from Address[IM01]:5701! This node is not requested endpoint: Address[192.168.70.73]:5701

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [/192.168.70.73:1961] lost. Reason: Explicit close

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:13 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.73:5702. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.73:5703. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.74:5702. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.74:5701. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.74:5703. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 1970 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM01]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.73:1970

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 5701 accepted socket connection from /192.168.70.73:1970

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM01]:5701 [openfire] Wrong bind request from Address[IM01]:5701! This node is not requested endpoint: Address[192.168.70.73]:5701

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:04:14 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [/192.168.70.73:1970] lost. Reason: Explicit close

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.73:5703. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.73:5702. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.74:5702. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.74:5701. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketConnector

INFO: [IM01]:5701 [openfire] Could not connect to: /192.168.70.74:5703. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 1971 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM01]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.73:1971

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 5701 accepted socket connection from /192.168.70.73:1971

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM01]:5701 [openfire] Wrong bind request from Address[IM01]:5701! This node is not requested endpoint: Address[192.168.70.73]:5701

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [/192.168.70.73:1971] lost. Reason: Explicit close

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:15 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM01]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.73:1972

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 5701 accepted socket connection from /192.168.70.73:1972

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 1972 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM01]:5701 [openfire] Wrong bind request from Address[IM01]:5701! This node is not requested endpoint: Address[192.168.70.73]:5701

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:04:16 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [/192.168.70.73:1972] lost. Reason: Explicit close

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM01]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.73:1973

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 5701 accepted socket connection from /192.168.70.73:1973

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 1973 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM01]:5701 [openfire] Wrong bind request from Address[IM01]:5701! This node is not requested endpoint: Address[192.168.70.73]:5701

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [/192.168.70.73:1973] lost. Reason: Explicit close

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:17 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM01]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.73:1974

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 5701 accepted socket connection from /192.168.70.73:1974

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 1974 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM01]:5701 [openfire] Wrong bind request from Address[IM01]:5701! This node is not requested endpoint: Address[192.168.70.73]:5701

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:04:18 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [/192.168.70.73:1974] lost. Reason: Explicit close

Feb 22, 2013 9:04:18 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM01]:5701 [openfire]

Members [1] {

      Member [IM01]:5701 this

}

Feb 22, 2013 9:04:18 AM com.hazelcast.impl.LifecycleServiceImpl

INFO: [IM01]:5701 [openfire] Address[IM01]:5701 is STARTED

Feb 22, 2013 9:04:18 AM com.hazelcast.impl.PartitionManager

INFO: [IM01]:5701 [openfire] Initializing cluster partition table first arrangement…

IM02 (With IM01 online and users can connect to 192.168.70.75):

Openfire 3.8.0 [Feb 22, 2013 9:09:36 AM]

Admin console listening at:

http://im:9090

https://im:9091

Starting Client Control Plugin

Starting Monitoring Plugin

Starting Hazelcast Clustering Plugin

Feb 22, 2013 9:09:43 AM com.hazelcast.config.ClasspathXmlConfig

INFO: Configuring Hazelcast from ‘hazelcast-cache-config.xml’.

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.AddressPicker

INFO: Resolving domain name ‘IM01’ to address(es): [192.168.70.71]

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.AddressPicker

INFO: Resolving domain name ‘IM02’ to address(es): [192.168.70.72, 192.168.70.74, 192.168.70.75]

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.AddressPicker

INFO: Interfaces is enabled, trying to pick one address matching to one of: [IM02/192.168.70.74]

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.AddressPicker

INFO: Prefer IPv4 stack is true.

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.AddressPicker

INFO: Picked Address[IM02]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true

Feb 22, 2013 9:09:45 AM com.hazelcast.system

INFO: [IM02]:5701 [openfire] Hazelcast Community Edition 2.5 (20130115) starting at Address[IM02]:5701

Feb 22, 2013 9:09:45 AM com.hazelcast.system

INFO: [IM02]:5701 [openfire] Copyright © 2008-2012 Hazelcast.com

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.LifecycleServiceImpl

INFO: [IM02]:5701 [openfire] Address[IM02]:5701 is STARTING

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM02]:5701 [openfire] Connecting to possible member: Address[192.168.70.74]:5702

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM02]:5701 [openfire] Connecting to possible member: Address[192.168.70.74]:5701

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM02]:5701 [openfire] Connecting to possible member: Address[192.168.70.73]:5703

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM02]:5701 [openfire] Connecting to possible member: Address[192.168.70.74]:5703

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM02]:5701 [openfire] Connecting to possible member: Address[192.168.70.73]:5702

Feb 22, 2013 9:09:45 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM02]:5701 [openfire] Connecting to possible member: Address[192.168.70.73]:5701

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1510 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1507 accepted socket connection from /192.168.70.74:5701

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM02]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.74:1507

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 5701 accepted socket connection from /192.168.70.74:1507

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM02]:5701 [openfire] Wrong bind request from Address[IM02]:5701! This node is not requested endpoint: Address[192.168.70.74]:5701

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [/192.168.70.74:1507] lost. Reason: Explicit close

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.74]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:45 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketConnector

INFO: [IM02]:5701 [openfire] Could not connect to: /192.168.70.74:5703. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketConnector

INFO: [IM02]:5701 [openfire] Could not connect to: /192.168.70.74:5702. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketConnector

INFO: [IM02]:5701 [openfire] Could not connect to: /192.168.70.73:5703. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketConnector

INFO: [IM02]:5701 [openfire] Could not connect to: /192.168.70.73:5702. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1516 accepted socket connection from /192.168.70.74:5701

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM02]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.74:1516

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 5701 accepted socket connection from /192.168.70.74:1516

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM02]:5701 [openfire] Wrong bind request from Address[IM02]:5701! This node is not requested endpoint: Address[192.168.70.74]:5701

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [/192.168.70.74:1516] lost. Reason: Explicit close

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.74]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1520 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:46 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketConnector

INFO: [IM02]:5701 [openfire] Could not connect to: /192.168.70.74:5702. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketConnector

INFO: [IM02]:5701 [openfire] Could not connect to: /192.168.70.74:5703. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketConnector

INFO: [IM02]:5701 [openfire] Could not connect to: /192.168.70.73:5702. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketConnector

INFO: [IM02]:5701 [openfire] Could not connect to: /192.168.70.73:5703. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1523 accepted socket connection from /192.168.70.74:5701

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM02]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.74:1523

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 5701 accepted socket connection from /192.168.70.74:1523

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1524 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM02]:5701 [openfire] Wrong bind request from Address[IM02]:5701! This node is not requested endpoint: Address[192.168.70.74]:5701

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [/192.168.70.74:1523] lost. Reason: Explicit close

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:47 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.74]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1525 accepted socket connection from /192.168.70.74:5701

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM02]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.74:1525

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 5701 accepted socket connection from /192.168.70.74:1525

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM02]:5701 [openfire] Wrong bind request from Address[IM02]:5701! This node is not requested endpoint: Address[192.168.70.74]:5701

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.74]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [/192.168.70.74:1525] lost. Reason: Explicit close

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1526 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:48 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1527 accepted socket connection from /192.168.70.74:5701

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM02]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.74:1527

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1528 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 5701 accepted socket connection from /192.168.70.74:1527

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM02]:5701 [openfire] Wrong bind request from Address[IM02]:5701! This node is not requested endpoint: Address[192.168.70.74]:5701

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [/192.168.70.74:1527] lost. Reason: Explicit close

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.74]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:49 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1529 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM02]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.74:1530

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 5701 accepted socket connection from /192.168.70.74:1530

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 1530 accepted socket connection from /192.168.70.74:5701

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:50 AM com.hazelcast.impl.TcpIpJoiner

INFO: [IM02]:5701 [openfire]

Members [1] {

      Member [IM02]:5701 this

}

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM02]:5701 [openfire] Wrong bind request from Address[IM02]:5701! This node is not requested endpoint: Address[192.168.70.74]:5701

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [/192.168.70.74:1530] lost. Reason: Explicit close

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:50 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [Address[192.168.70.74]:5701] lost. Reason: java.io.EOFException[null]

Feb 22, 2013 9:09:50 AM com.hazelcast.impl.LifecycleServiceImpl

INFO: [IM02]:5701 [openfire] Address[IM02]:5701 is STARTED

Feb 22, 2013 9:09:50 AM com.hazelcast.impl.PartitionManager

INFO: [IM02]:5701 [openfire] Initializing cluster partition table first arrangement…

Feb 22, 2013 9:10:17 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM02]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.71:1988

Feb 22, 2013 9:10:17 AM com.hazelcast.nio.ConnectionManager

INFO: [IM02]:5701 [openfire] 5701 accepted socket connection from /192.168.70.71:1988

Feb 22, 2013 9:10:17 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:10:17 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:10:17 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM02]:5701 [openfire] Wrong bind request from Address[IM01]:5701! This node is not requested endpoint: Address[192.168.70.74]:5701

Feb 22, 2013 9:10:17 AM com.hazelcast.nio.Connection

INFO: [IM02]:5701 [openfire] Connection [/192.168.70.71:1988] lost. Reason: Explicit close

Feb 22, 2013 9:12:17 AM com.hazelcast.nio.SocketAcceptor

INFO: [TSG1IM02]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.71:1995

Feb 22, 2013 9:12:17 AM com.hazelcast.nio.ConnectionManager

INFO: [TSG1IM02]:5701 [openfire] 5701 accepted socket connection from /192.168.70.71:1995

Feb 22, 2013 9:12:17 AM com.hazelcast.nio.SocketPacketWriter

INFO: [TSG1IM02]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:12:17 AM com.hazelcast.nio.SocketPacketReader

INFO: [TSG1IM02]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:12:17 AM com.hazelcast.nio.ConnectionManager

WARNING: [TSG1IM02]:5701 [openfire] Wrong bind request from Address[TSG1IM01]:5701! This node is not requested endpoint: Address[192.168.70.74]:5701

Feb 22, 2013 9:12:17 AM com.hazelcast.nio.Connection

INFO: [TSG1IM02]:5701 [openfire] Connection [/192.168.70.71:1995] lost. Reason: Explicit close

After IM02 was brought online, IM01 had the following multiple times:

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 1985 accepted socket connection from /192.168.70.73:5701

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.SocketAcceptor

INFO: [IM01]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.73:1985

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.ConnectionManager

INFO: [IM01]:5701 [openfire] 5701 accepted socket connection from /192.168.70.73:1985

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.SocketPacketWriter

INFO: [IM01]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.SocketPacketReader

INFO: [IM01]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.ConnectionManager

WARNING: [IM01]:5701 [openfire] Wrong bind request from Address[IM01]:5701! This node is not requested endpoint: Address[192.168.70.73]:5701

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [/192.168.70.73:1985] lost. Reason: Explicit close

Feb 22, 2013 9:09:20 AM com.hazelcast.nio.Connection

INFO: [IM01]:5701 [openfire] Connection [Address[192.168.70.73]:5701] lost. Reason: java.io.EOFException[null]

My apologies for the long post and all the log information. But, as you can see, neither server really connects with the other on the cluster.

How can I resolve the red above and allow redundant connections from 192.168.70.73 &192.168.70.74 and allow the users to connect to a single IP: 192.168.70.75.

If Microsoft Network Load Balancing isn’t supported, can you use the public IP noted in the config to allow all users to connect to the cluster HazelCast creates?

Lastly, how can I enable IPv4 since it is preferred?

Thank you very much for your support,

Johnathan

2

Can you change your config to look like this? Hazelcast doesn’t care about the public IP address that users connect on.

5701

192.168.70.71:5701

192.168.70.72:5701

For what it is worth, I am not following how your networking is setup with two NICs apparently on the same subnet. It also looks like your hostnames are resolving to multiple IPs, which might be confusing to Hazelcast.

3

@David,

I have done as you’ve suggested.

The two servers do see each other. I can still only connect to 192.168.70.75 only when IM01 is online. Even though IM02 is still functional.

With each server having two NICs, the idea was to allow connectivity from the outside of the servers to the public LAN on one set and allow clustering on the 2nd set of NIC.

How can I make a singular address (192.168.70.75) to be used to connect to both servers? I would think this would have to be configured in the config file.

When IM01 was brought online with the new config:

Openfire 3.8.0 [Feb 22, 2013 10:09:44 AM]

Admin console listening at:

http://im:9090

https://im:9091

Starting Client Control Plugin

Starting Monitoring Plugin

Starting Hazelcast Clustering Plugin

Feb 22, 2013 10:09:50 AM com.hazelcast.config.ClasspathXmlConfig

INFO: Configuring Hazelcast from ‘hazelcast-cache-config.xml’.

Feb 22, 2013 10:09:51 AM com.hazelcast.impl.AddressPicker

INFO: Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [192.168.70.71, 192.168.70.72]

Feb 22, 2013 10:09:51 AM com.hazelcast.impl.AddressPicker

INFO: Prefer IPv4 stack is true.

Feb 22, 2013 10:09:51 AM com.hazelcast.impl.AddressPicker

INFO: Picked Address[192.168.70.71]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true

Feb 22, 2013 10:09:51 AM com.hazelcast.system

INFO: [192.168.70.71]:5701 [openfire] Hazelcast Community Edition 2.5 (20130115) starting at Address[192.168.70.71]:5701

Feb 22, 2013 10:09:51 AM com.hazelcast.system

INFO: [192.168.70.71]:5701 [openfire] Copyright © 2008-2012 Hazelcast.com

Feb 22, 2013 10:09:51 AM com.hazelcast.impl.LifecycleServiceImpl

INFO: [192.168.70.71]:5701 [openfire] Address[192.168.70.71]:5701 is STARTING

Feb 22, 2013 10:09:51 AM com.hazelcast.impl.TcpIpJoiner

INFO: [192.168.70.71]:5701 [openfire] Connecting to possible member: Address[192.168.70.72]:5701

Feb 22, 2013 10:09:52 AM com.hazelcast.nio.SocketConnector

INFO: [192.168.70.71]:5701 [openfire] Could not connect to: /192.168.70.72:5701. Reason: ConnectException[Connection refused: connect]

Feb 22, 2013 10:09:52 AM com.hazelcast.impl.TcpIpJoiner

INFO: [192.168.70.71]:5701 [openfire]

Members [1] {

      Member [192.168.70.71]:5701 this

}

Feb 22, 2013 10:09:52 AM com.hazelcast.impl.LifecycleServiceImpl

INFO: [192.168.70.71]:5701 [openfire] Address[192.168.70.71]:5701 is STARTED

Feb 22, 2013 10:09:52 AM com.hazelcast.impl.PartitionManager

INFO: [192.168.70.71]:5701 [openfire] Initializing cluster partition table first arrangement…

Feb 22, 2013 10:09:53 AM com.hazelcast.nio.SocketConnector

INFO: [192.168.70.71]:5701 [openfire] Could not connect to: /192.168.70.72:5701. Reason: ConnectException[Connection refused: connect]

When IM02 is brought online from IM01 log:

Feb 22, 2013 10:10:11 AM com.hazelcast.nio.SocketAcceptor

INFO: [192.168.70.71]:5701 [openfire] 5701 is accepting socket connection from /192.168.70.72:1803

Feb 22, 2013 10:10:11 AM com.hazelcast.nio.ConnectionManager

INFO: [192.168.70.71]:5701 [openfire] 5701 accepted socket connection from /192.168.70.72:1803

Feb 22, 2013 10:10:12 AM com.hazelcast.nio.SocketPacketWriter

INFO: [192.168.70.71]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 10:10:12 AM com.hazelcast.nio.SocketPacketReader

INFO: [192.168.70.71]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 10:10:18 AM com.hazelcast.cluster.ClusterManager

INFO: [192.168.70.71]:5701 [openfire]

Members [2] {

      Member [192.168.70.71]:5701 this

      Member [192.168.70.72]:5701

}

Feb 22, 2013 10:10:19 AM com.hazelcast.impl.PartitionManager

INFO: [192.168.70.71]:5701 [openfire] Re-partitioning cluster data… Immediate-Tasks: 271, Scheduled-Tasks: 0

IM02 log:

Openfire 3.8.0 [Feb 22, 2013 10:09:07 AM]

Admin console listening at:

http://im:9090

https://im:9091

Starting Client Control Plugin

Starting Monitoring Plugin

Starting Hazelcast Clustering Plugin

Feb 22, 2013 10:09:13 AM com.hazelcast.config.ClasspathXmlConfig

INFO: Configuring Hazelcast from ‘hazelcast-cache-config.xml’.

Feb 22, 2013 10:09:14 AM com.hazelcast.impl.AddressPicker

INFO: Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [192.168.70.71, 192.168.70.72]

Feb 22, 2013 10:09:14 AM com.hazelcast.impl.AddressPicker

INFO: Prefer IPv4 stack is true.

Feb 22, 2013 10:09:14 AM com.hazelcast.impl.AddressPicker

INFO: Picked Address[192.168.70.72]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true

Feb 22, 2013 10:09:14 AM com.hazelcast.system

INFO: [192.168.70.72]:5701 [openfire] Hazelcast Community Edition 2.5 (20130115) starting at Address[192.168.70.72]:5701

Feb 22, 2013 10:09:14 AM com.hazelcast.system

INFO: [192.168.70.72]:5701 [openfire] Copyright © 2008-2012 Hazelcast.com

Feb 22, 2013 10:09:14 AM com.hazelcast.impl.LifecycleServiceImpl

INFO: [192.168.70.72]:5701 [openfire] Address[192.168.70.72]:5701 is STARTING

Feb 22, 2013 10:09:14 AM com.hazelcast.impl.TcpIpJoiner

INFO: [192.168.70.72]:5701 [openfire] Connecting to possible member: Address[192.168.70.71]:5701

Feb 22, 2013 10:09:14 AM com.hazelcast.nio.ConnectionManager

INFO: [192.168.70.72]:5701 [openfire] 1803 accepted socket connection from /192.168.70.71:5701

Feb 22, 2013 10:09:14 AM com.hazelcast.nio.SocketPacketWriter

INFO: [192.168.70.72]:5701 [openfire] Writer started with SymmetricEncryption

Feb 22, 2013 10:09:14 AM com.hazelcast.nio.SocketPacketReader

INFO: [192.168.70.72]:5701 [openfire] Reader started with SymmetricEncryption

Feb 22, 2013 10:09:21 AM com.hazelcast.cluster.ClusterManager

INFO: [192.168.70.72]:5701 [openfire]

Members [2] {

      Member [192.168.70.71]:5701

      Member [192.168.70.72]:5701 this

}

Feb 22, 2013 10:09:23 AM com.hazelcast.impl.LifecycleServiceImpl

INFO: [192.168.70.72]:5701 [openfire] Address[192.168.70.72]:5701 is STARTED

Would using the hostname with the two cluster NIC create what I am wanting? Use 192.168.70.73 & 192.168.70.74 instead of what it is now?

Thank you for your time and support,

Johnathan

Message was edited by: Johnathan

4

I’ve never used Microsoft NLB, but in my UNIX environment I have my load balancer configured to do health checks against Openfire so either it only routes users to an Openfire isntance that is running, or in your case, would only run the VIP (.75) on a system were Openfire is running.

Hazelcast does not have the ability to manage virtual IP addresses on your system. I would expect there is a way with NLB to have it monitor or be dependent upon a running service and move the IP address when the service is down. So, I’d start by looking at NLB documentation and figuring out how that works.

I think your networking is screwy too, since you have both NICs on each box in the same subnet. Assuming they’re on different VLANs or virtual switches inside VMware, you really want them on different subnets. That said, you really don’t need a separate NIC in this configuration, so it’s probably just causing more problems than it is solving.

5

@David,

With the two NIC setup, we’ve used this with our other servers and have not had issues. Given, none of those servers are using anything other than Microsoft software.

I could remove the 2nd NIC and should work just fine I suppose.

So, just to make this clear, HazelCast clustering is really only used to replicate an OpenFire server’s data, nothing more, correct?

It can’t allow a singular IP for everyone to connect to, correct?

I would think that allowing a singular IP for folks to connect to would be apart of the clustering with HazelCast, especially if you bring into count that you can use 10,000 nodes. Having to manage all the IP’s, etc. in a large multi-national environment would be hectic at least. Given, they would probably use some commercial IM server anyway.

If there is any other advice, please do.

Thank you for your time and support,

Johnathan

6

Yes, Hazelcast is only for clustering the Openfire application itself. In general, Openfire/Hazelcast doesn’t run with administrative privileges to be able to add/remove IP addresses on the system anyway.

If you want a singular IP, you will need to configure some sort of clustering/load balancing software - Maybe NLB can do it if it is configured to ‘monitor’ the Openfire service and move IPs when it dies. We have a separate load balancer we route XMPP through which does XMPP tests to monitor the availability of each Openfire node.

7

@David,

As far as the separate software, that is what Microsoft NLB was for, I just have to figure out how to make it work with OpenFire. We use Microsoft NLB for using a singular IP or DNS name to allow 1000’s of workstations to connect to a web page. So I know my cluster is setup correctly.

On your UNIX platform… I assume you have a singular IP that your users connect to, correct? If so, how did you configure the HazelCast config file to use that IP? If not, then I am just stuck until I can figure it out with using multiple IPs for users to connect to.

Thanks again,

Johnathan

8

How does the web service work with NLB if IIS is shutdown, but the system is still up. Web services might be a little different, since many web browsers will try each IP returned by DNS, rather than the first one then give up. Either way, IIS might have hooks into NLB that disable that node when IIS is down.

In my environment, Hazelcast knows nothing about the single virtual IP users connect on - it doesn’t need to. The load balancers have that IP, and route XMPP requests (as well as other Openfire related stuff like admin console) to the backend nodes. There is zero configuration in either Openfire or Hazelcast related to our virtual IP - That’s all handled on the networking and OS end.

9

@David,

The IIS service is running on at least two servers at a time. Thus, if one server goes offline, the other server is still running. The NLB shared the load and connection between the two servers. Hence, the singular cluster IP that they share to allow a singular IP for everyone to connect to.

Since I’ve configured Microsoft NLB, it should be able to work… but it’s not…

Thanks again,

Johnathan

10

DIdn’t really answer the question - When IIS is down on one node, how does NLB know not to send users to that node? Basically, that is what you need to accomplish with Openfire. Again, NLB might not actually do anything and rely on HTTP reconnecting to another box.

11

@David,

How exactly does NLB work, as far as I know it monitors the two host IPs that you assign to the cluster and manages it from there. IIS then uses the cluster IP and all is well.

I was hoping that OpenFire would be similar… you point the HazelCast config to the Cluster IP and there you go. But alas, that is not how it is.

Thanks again,

Johnathan

12

@David,

Another note, when creating the NLB Cluster, you assign what ports to monitor. In this case instead of just monitoring port 80,IO allowed the entire rage from 0 to 65536.

Thanks again,

Johnathan

13

You probably want it to monitor port 5222 instead of 80.

14

I don’t know much about clustering and load balancing, but aren’t you talking about different things here?

I believe Johnathan want to achieve that his NLB servers would choose to what Openfire node forward new connections to balance the load. Say IM01 already has 1000 connections and IM02 has only 500, so NLB Cluster will forward new xmpp connection to IM02. A long as i remember the theory about IIS load balancing, it could be one IIS server, but many other nodes could be serving a web page and those nodes would be on the NLB servers. Or maybe two IIS servers. Maybe i’m wrong. Anyway, i’m not sure if and how NLB can monitor Openfire connections and load. Simply by counting number of connections to 5222 port? It can miscalculate something as NLB is probably not intended to use with other 3d party services.

David is talking about a pure clustering solution. When IM01 is serving all the connections and IM02 just sits there in reserve and only serves the connections when IM01 dies. Can’t talk for Hazelcast developers, but maybe it’s the only setup they were aiming for.

On a side note. There is another way to handle the load. By using Openfire Connection Managers. You can have a number of CM nodes, which congregate incoming connections and then forward them to the main Openfire server as a single connection.

15

@wroot,

I am lookig for a two-fold solution:

  1. a load balancing solution to transfer the connections across both servers.

  2. a redunant connection where if one server dies, no proble, the already running server runs with all of the connections.

I was simply using IIS as a reference as we already use it to display web pages.

IIS and Microsoft NLB are two different things/programs.

Right now, as it stands, HazelCast is running and sees both servers. However, ALL connections are on one server, not both of them. With that, I have no redundency. HazelCast, I suppose, is supposed to replicate the data between servers. So right now, if my IM01 goes down… so does the IM service. I then have to redirect everyone to the second server IP. This can be messy.

Does that make more sense?

I am not familiar with what a ‘Connection Manager’ is or what it can do for me. Can you explain further?

Thank you,

Johnathan

16

There are certainly two approaches to clustering anything - Use a load balanacer, or run something on the nodes themselves to distribute the traffic. In either case, you need a reliable way to determine if you should route traffic to a specific node or not. I don’t know anything specifically about NLB, but if I was doing it on UNIX I’d monitor for the existance of a process, or tcp port, to determine if that system was ‘alive’ enough to get connections.

We run in an active/active two-node config with a load balancer in front - It does XMPP logins to each node and doesn’t route connections to that system if authentication fails. Any type of load sharing configuration is useless if you don’t have a reliable way to figure out if an instance is usable or not.

Like I said, I don’t know NLB from a hole in the ground, so maybe it’s as simple as telling it to ‘monitor’ port 5222 instead of 80. Or maybe it just works with MS services. The problem Johnathan appeared to be having was that NLB was cluseless if the Openfire service was actually running on each system, so would route traffic to that node even if OF was down. At least, that’s what I got out of the discussion

17

Even with a real load balancer, users will be disconnected then will reconnect to another available Openfire instance in the event of one node going down. I’m not aware of a mechanism to transition one XMPP connection to another Openfire instance transparent to the client - XMPP has a persistent connection to the JVM, rather than a higher level ‘session’ like you have with HTTP.

Connection Manager is a separate Java process which functions as an XMPP proxy. Not sure how it would help in your situation - You’d need to load balance connections to the connection manager proxy (in case the proxy itself goes down), plus you’d have to be able to failover the proxy->Openfire connection between nodes too. Seems more complicated than what you have right now.

18

@All,

NLB current is monitoring all ports from 0 to 65536. I could limit it to ports 5222-5223 and see if that helps.

My thinking is that it would work even not limiting the ports though.

It could be that only Microsoft stuff works with it, but that would defeat the purpose if MS doesn’t have what you are trying to Load Balance, like an IM server for instance. BTW, looking for free solutions… not paid for ones. So Sharepoint, Lynk and others are useless ATM.

How I believe Microsoft NLB works is this:

You have one IP on one server, another IP on another server. You link them to a cluster IP. In this case:

192.168.70.73 <–> 192.168.70.74 = 192.168.70.75

I know that using IIS with it, if one server goes down, NLB then transfers all traffic to the active server… thus a redundant connection with 192.168.70.75 being a singular IP that others can connect to.

Hope this helps.

@David,

Thanks for clearing that up with Connection Manager plugins.

Thanks for the help,

Johnathan

19

BTW, for more information on Microsoft Network Load Balancing (NLB):

http://technet.microsoft.com/en-us/library/cc758834(v=ws.10).aspx

20

@David,

To answer your question of how NLB detects failed connections:

Q. How Does NLB Detect a Server Failure?

A. While servicing client requests, each NLB Cluster host emits heartbeats to the other hosts in the cluster. If a host fails and stops emitting heartbeats, then after a default time period of five seconds, the remaining hosts in the cluster undergo a process called convergence to remove the failed host from the cluster and have new client connection requests mapped to remaining hosts in the cluster. So, new connections made by the client get handled by one of the remaining hosts in the cluster.