Hello! We have some issues that we would like to consult with you about a Project developed using openfire. Especially, clustering. Let’s talk about what we did. We have already used openfire to chat with our staff 400 persons, and we have already experienced this for four years in amateur way. We consider to sell the application. The companies that we will sell the aplication have between 1000 and 2000 users. Hovewer, the plugins and servers we use are not seen enough for a single openfire. That’s why we need clustering to handle the load which is required. Also, we have a touble with the MUC. When we use clustering, users receive the following package when sending messages to groups:
<error code="406" type="modify"><not-acceptable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error>
Let’s talk about the method our using clustering. First of all, we dockerize openfire. We share the document as an open source at this address. Dockerfile is designed in a very simple way. All thing that should be done is downloaded the specified version file of openfire and install the package.
The last version of dockerized openfire is 4.4.2. The package is stored on hub.docker.com with the name abdurrahmanekr/openfire:4.4.2.
To make clustering, we create multiple openfire services with docker-compose and start them with docker-compose up -d. And all openfire services are connected to a single volume. Our example docker-compose.yml is followed as:
version: '2'
services:
openfire1:
restart: always
depends_on:
- db
image: abdurrahmanekr/openfire:4.4.2
ports:
- "17070:7070"
- "19090:9090"
volumes:
- 'openfire-data:/var/lib/openfire'
- 'openfire-plugins:/usr/share/openfire/plugins'
openfire2:
restart: always
depends_on:
- db
image: abdurrahmanekr/openfire:4.4.2
ports:
- "17071:7070"
- "19091:9090"
volumes:
- 'openfire-data:/var/lib/openfire'
- 'openfire-plugins:/usr/share/openfire/plugins'
# DATABASE
db:
image: mysql:5.7.26
ports:
- "3305:3306"
volumes:
- '/var/mysql:/var/lib/mysql'
command: --default-authentication-plugin=mysql_native_password
environment:
MYSQL_ROOT_PASSWORD: '1234'
MYSQL_DATABASE: 'openfire'
volumes:
openfire-data:
openfire-plugins:
As you can see, two openfire services are running on the same network. We link the openfire services with the following setting hazelcast-local-config.xml:
<?xml version="1.0" encoding="UTF-8"?>
<hazelcast xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.hazelcast.com/schema/config
http://www.hazelcast.com/schema/config/hazelcast-config-3.12.xsd">
<group>
<name>openfire</name>
<password>openfire</password>
</group>
<network>
<port auto-increment="true" port-count="100">5701</port>
<outbound-ports>
<ports>0</ports>
</outbound-ports>
<!-- The following enables multicast discovery of cluster members
See http://docs.hazelcast.org/docs/3.12/manual/html-single/index.html#discovering-members-by-multicast
-->
<!--
<join>
<multicast enabled="true">
<multicast-group>224.2.2.3</multicast-group>
<multicast-port>54327</multicast-port>
</multicast>
<tcp-ip enabled="false"/>
</join>
-->
<!-- The following enables TCP/IP based discovery of cluster members
See http://docs.hazelcast.org/docs/3.12/manual/html-single/index.html#discovering-members-by-tcp
-->
+ <join>
+ <multicast enabled="false"/>
+ <tcp-ip enabled="true">
+ <member>openfire1:5701</member>
+ <member>openfire2:5701</member>
+ </tcp-ip>
+ </join>
<interfaces enabled="false">
<interface>10.10.1.*</interface>
</interfaces>
<ssl enabled="false"/>
<socket-interceptor enabled="false"/>
<symmetric-encryption enabled="false">
<!--
encryption algorithm such as
DES/ECB/PKCS5Padding,
PBEWithMD5AndDES,
AES/CBC/PKCS5Padding,
Blowfish,
DESede
-->
<algorithm>PBEWithMD5AndDES</algorithm>
<!-- salt value to use when generating the secret key -->
<salt>thesalt</salt>
<!-- pass phrase to use when generating the secret key -->
<password>thepass</password>
<!-- iteration count to use when generating the secret key -->
<iteration-count>19</iteration-count>
</symmetric-encryption>
</network>
</hazelcast>
We use haproxy to access and balance Openfire with BOSH from outside. The reason for using Haproxy, the sticky session feature appeares.We need the sticky session feature thats why if we had balanced without sticky session feature, the user would always get 404 error for http-bind. As far as we know Openfire cannot be managed session with hazelcast.
The haproxy.cfg file used for balance is as follows:
global
maxconn 4096
pidfile ~/tmp/haproxy-queue.pid
defaults
log global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
mode http
timeout connect 300000
timeout client 300000
timeout server 300000
maxconn 2000
option redispatch
retries 3
option httpclose
option httplog
option forwardfor
option httpchk HEAD / HTTP/1.0
backend openfire_restapi
balance roundrobin
mode http
server openfire_restapi1 localhost:19090 check
server openfire_restapi2 localhost:19091 check
backend openfire_http_bind
balance roundrobin
mode http
http-check expect status 404
cookie SERVERID insert indirect nocache
server ha1 localhost:17070 check cookie ha1
server ha2 localhost:17071 check cookie ha2
frontend openfire_balance
bind *:10071 # HAPROXY PORT
mode http
acl url_http_bind path_beg /http-bind
acl url_chatservice path_beg /plugins/
use_backend openfire_restapi if url_chatservice
use_backend openfire_http_bind if url_http_bind
As you can see, balance can be provide with the haproxy 10071 port. We have an nginx server with port 80 out of which can access Haproxy. The settings of the nginx server is as follows:
upstream openfire_upstream {
# Only one machine
# Because haproxy is in the back-end
server localhost:10071;
}
server {
listen 80;
server_name exampleopenfire.com;
location /plugins/ {
# Openfire Services
proxy_pass http://openfire_upstream/plugins/;
}
location /http-bind/ {
# Openfire BOSH Connection
proxy_pass http://openfire_upstream/http-bind/;
}
}
Our Architecture
With the structure, we are able to run the personal conversation (<message type="chat") properly and any problem is not occured. When we tried to make groupchat conversations, it seems there is not any problem, but when the cookie value (SERVERID) of the user is changed, we recognize the error package below.
Regardless of the cookies, we form a group and we can chat with each other. We removed one of our staff from this group and added again (with RESTAPI plugin). One of our staff changes his SESSIONID from ha1 to ha2. The one’s messages are not able to transmitted to the group chat room and he cannot write any message to the group chat room. While the one changes his SESSIONID from ha2 to ha1 again, his messages are able to transmitted to the group chat room and he can write a message to the group chat room. That means it work. This situation is experienced by all staff in the group chat. When the group is assembled at first, they all are able to chat without any problems. But the staff whose cookies are changed has encountered the error. In summary, the messages in the group are not received and any messages are not able to write to the group chat room. (not-acceptable) error
<error code="406" type="modify"><not-acceptable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error>
NOT
SESSIONID is equal ha1, it assigns a request to openfire1 service. SESSIONID is equal ha2, it assigns a request to openfire2 service
Only BOSH is used on Openfire. (not using WebSocket).
Any configuration is not done.
Only RESTAPI plugin and Hazelcast plugin are used. The RESTAPI plugin is used to create a group, add member and .
Kind regards.