How to optimize performance for openfire 4.0.1

Hi,

Is there any idea to optimize performance for openfire 4.0.1?

I’ve got 800+ users, and 50+ groups shared for all users in the Roster.

Every morning when 300+ offline users log on, some user cannot log on before they tried several times.

I’v changed ulimit -n to 65535 on the Openfire server,

and add “ -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.EPollSelectorProvider” to OPENFIRE_OPTS in /etc/init.d/openfire.

also changed sysctl: net.ipv4.ip_local_port_range = 1024 65000

It seems useless.

This is some info:

Openfire:4.0.1

OS: RHEL6 x86_64

Java:1.8.0_45 Oracle Corporation – Java HotSpot™ 64-Bit Server VM

Java Memory: 1127.00 MB of 54613.50 MB (2.1%) used

Thanks a lot!

Need to use CM(connection manager) ?

800 Users should be not a problem.

Do you use LDAP? Maybe is the DB the bottleneck?

Yes.

Users are from LDAP.

I will check DB connection next morning, the problem happened when all users logging on every morning 8:30-09:00 .

Thank you!

Try also to set more cache (it seems that you have enough memory)

e.g.

cache.username2roster.size 104857600

cache.group.size 40000000

cache.userCache.size 35000000

cache.lastActivity.size 1500000

cache.offlinePresence.size 1500000

cache.userGroup.size 104857600

cache.groupMeta.size 10485760

cache.listsCache.size 52428800

Thanks!

After set more cache and restart openfire service, The speed of users loging on is faster now.

But 50+ users are still hard to logon when 200+ users loging on at 08:30-09:00.

Spark error log shows “Connection failed. No response from server.”

The database query was very slow:

Query

Count

Total Time

  1. Avg. Time
    SELECT name, list FROM ofPrivacyList WHERE username=? AND isDefault=?

8,267

4,774

0

Should I do something for DB (MySQL5.1) ?

if your using AD, than that usernames get cached, and authentication happens with ldap, and not the database…however, the database does store things like offline messages, private data, rosters, etc.

A few things to try is to make sure pool connections are enabled ldap.connectionPoolEnabled =true

also i bet your AD has more than 1000 objects, so you’ll want to set ldap.pagedResultsSize = 1000

Also another thing you might be running into is a locking issue on reads with your DB. I haven’t tested this, but you might want to look into setting your isolation level to read committed.

also, I’m curious about your environment. are you running vmware?

Thanks speedy.

I set ldap options on my server:

ldap.connectionPoolEnabled & ldap.pagedResultsSize

I think you are right. My DB may have a locking issue on reads.

I tried to update my.cnf as below:

#Optimize for Openfire

skip-locking

skip-name-resolve

max_connections=500

back_log=500

#wait_timeout=1800

max_allowed_packet=4M

sort_buffer_size=32M

read_buffer_size=4M

read_rnd_buffer_size=16M

myisam_sort_buffer_size=64M

#thread_cache_size=128

#query_cache_size=32M

thread_concurrency=16

max_heap_table_size=64M

join_buffer_size=16M

key_buffer_size=512M

innodb_additional_mem_pool_size=4M

tmp_table_size=64M

Here is Database Query Statistics:

SELECT Query Statistics

Total # of operations
1,477,237
Total time for all operations (ms)
873,800
Average time for each operation (ms)
0.59
Operations per second
9.14
Most common SQL queries

Query
Count
Total Time
Avg. Time
SELECT name, list FROM ofPrivacyList WHERE username=? AND isDefault=?
595,322
340,765
0
SELECT bytes from ofRRDs where id = ?
279,820
125,853
0
SELECT groupName from ofGroupProp where name=? AND propValue LIKE ?
94,295
64,986
0
SELECT name, propValue FROM ofUserProp WHERE username=?
71,826
24,379
0
SELECT DISTINCT serviceID FROM ofPubsubNode WHERE serviceID=?
64,014
20,774
0
SELECT bookmarkType, bookmarkName, bookmarkValue, isGlobal FROM ofBookmark WHERE bookmarkID=?
62,595
21,490
0
SELECT bookmarkType, name FROM ofBookmarkPerm WHERE bookmarkID=?
62,595
21,437
0
SELECT name, propValue FROM ofBookmarkProp WHERE bookmarkID=?
42,809
15,848
0
SELECT privateData FROM ofPrivate WHERE username=? AND name=? AND namespace=?
39,818
20,548
0
SELECT groupName from ofGroupProp WHERE name=? AND propValue=?
24,681
17,711
0
SELECT jid, rosterID, sub, ask, recv, nick FROM ofRoster WHERE username=?
24,597
14,535
0
SELECT name, propValue FROM ofGroupProp WHERE groupName=?
19,268
8,551
0
SELECT isExternal FROM ofConversation WHERE conversationID=?
15,483
3,698
0
SELECT id FROM ofID WHERE idType=?
15,247
5,702
0
SELECT room, isExternal, startDate, lastActivity, messageCount FROM ofConversation WHERE conversationID=?
8,263
2,964
0
SELECT bareJID, jidResource, nickname, joinedDate, leftDate FROM ofConParticipant WHERE conversationID=? ORDER BY joinedDate
8,263
44,724
5
SELECT bookmarkID from ofBookmark
6,955
2,472
0
SELECT ofRosterGroups.rosterID, groupName FROM ofRosterGroups INNER JOIN ofRoster ON ofRosterGroups.rosterID = ofRoster.rosterID WHERE username=? ORDER BY ofRosterGroups.rosterID, rank
5,843
5,587
0
SELECT sessionID, transcript, startTime FROM fpSession WHERE workgroupID=? AND startTime>? AND transcript IS NOT NULL ORDER BY startTime
5,386
2,491
0
SELECT name, startTime, endTime FROM ofUserFlag WHERE username=? AND name=?
4,586
1,964
0

INSERT Query Statistics

Total # of operations
52,118
Total time for all operations (ms)
32,852
Average time for each operation (ms)
0.63
Operations per second
0.32
Most common SQL queries

Query
Count
Total Time
Avg. Time
INSERT INTO ofConParticipant(conversationID, joinedDate, bareJID, jidResource, nickname) VALUES (?, ?, ?, ?, ?)
28,929
16,160
0
INSERT INTO ofConversation(conversationID, room, isExternal, startDate, lastActivity, messageCount) VALUES (?, ?, ?, ?, ?, 0)
14,466
5,399
0
INSERT INTO ofOffline (username, messageID, creationDate, messageSize, stanza) VALUES (?, ?, ?, ?, ?)
3,893
1,570
0
INSERT INTO ofMessageArchive(messageID, conversationID, fromJID, fromJIDResource, toJID, toJIDResource, sentDate, body, stanza) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
2,762
8,818
3
INSERT INTO ofPresence(username, offlinePresence, offlineDate) VALUES(?, ?, ?)
2,050
899
0
INSERT INTO ofRosterGroups (rosterID, rank, groupName) VALUES (?, ?, ?)
11
4
0
INSERT INTO ofRoster (username, rosterID, jid, sub, ask, recv, nick) VALUES (?, ?, ?, ?, ?, ?, ?)
5
2
0
INSERT INTO ofPrivacyList (username, name, isDefault, list) VALUES (?, ?, ?, ?)
1
0
0
INSERT INTO ofSecurityAuditLog(msgID, username, entryStamp, summary, node, details) VALUES(?, ?, ?, ?, ?, ?)
1
0
0

UPDATE Query Statistics

Total # of operations
47,527
Total time for all operations (ms)
55,480
Average time for each operation (ms)
1.17
Operations per second
0.29
Most common SQL queries

Query
Count
Total Time
Avg. Time
UPDATE ofRRDs SET bytes = ?, updatedDate=? WHERE id = ?
26,930
25,319
0
UPDATE ofID SET id=? WHERE idType=? AND id=?
15,247
4,566
0
UPDATE ofConversation SET lastActivity=?, messageCount=? WHERE conversationID=?
2,758
7,554
2
UPDATE ofConParticipant SET leftDate=? WHERE conversationID=? AND bareJID=? AND jidResource=? AND joinedDate=?
1,819
17,741
9
UPDATE ofBookmarkProp SET propValue=? WHERE name=? AND bookmarkID=?
716
265
0
UPDATE ofPrivacyList SET isDefault=?, list=? WHERE username=? AND name=?
52
32
0
UPDATE ofRoster SET sub=?, ask=?, recv=?, nick=? WHERE rosterID=?
4
3
0
UPDATE ofProperty SET propValue=? WHERE name=?
1
0
0

DELETE Query Statistics

Total # of operations
72,325
Total time for all operations (ms)
64,238
Average time for each operation (ms)
0.89
Operations per second
0.45
Most common SQL queries

Query
Count
Total Time
Avg. Time
DELETE FROM ofPresence WHERE username=?
44,858
20,839
0
DELETE FROM ofMessageArchive WHERE conversationID=?
8,263
32,434
3
DELETE FROM ofConParticipant WHERE conversationID=?
8,263
7,154
0
DELETE FROM ofConversation WHERE conversationID=?
8,263
2,589
0
DELETE FROM ofGojaraStatistics WHERE messageDate < ?
1,346
596
0
DELETE FROM ofOffline WHERE username=?
789
613
0
DELETE ofPubsubItem FROM ofPubsubItem LEFT JOIN (SELECT id FROM ofPubsubItem WHERE serviceID=? AND nodeID=? ORDER BY creationDate DESC LIMIT ?) AS noDelete ON ofPubsubItem.id = noDelete.id WHERE noDelete.id IS NULL AND ofPubsubItem.serviceID = ? AND nodeID = ?
539
12
0
DELETE FROM ofRosterGroups WHERE rosterID=?
4
1
0

I will monitor again next monday morning…

And my Openfire was installed on HP DL380 Gen8 Server, not running vmware.

max_connections=500 is very high. For every established connection Openfire and MySql will allocate memory (expensive operation) and you may get OutOfMemory errors. Monitor the number of connections with netstat and set this a bit higher. Openfire may better wait for a connection out of the pool instead of creating a new one which may take longer.

Permanently monitoring the number of established DB connections wouldn’t hurt.

Is the OF property database.defaultProvider.maxConnection still set to 10?

The System Properties “database.defaultProvider.maxConnections” default was 25, I changed it to 250.

I restart OF, but “Database Properties” still shows: 0 (active), 6 (available), 25 (max), not 250 (max).

So I think “database.defaultProvider.maxConnections” doesn’t work.

I need to change it back to 25?

And my.cnf max_connections=50 will be enough ?

Best thing is to run a script every minute to log the date together with the output of ‘netstat -an|grep ESTABLISHED|grep -c 3306’ (replace 3306 with the db socket port). So you’ll get a log file which shows the nr of established (idle or active) db connections. If you hit there a limit (eg 25) for a longer time then you may consider increasing the nr. If you never reach more than 5 connection even under heavy load you may set the limit to 10.

If you monitor also the other TCP ports you can correlate the nr. of http-bind, xmpp and db connections.

Where do you set the cache properties?

Openfire -->Server–>Server Manager–>System Properties

I think I’ve got the problem’s point:

We use ldap auth for users, and we use “Contact List (Roster) Sharing” to share all dept group for all users.

The problem is we have 800+ users, and 100+ dept groups, that means everyone’s Roster shows 100+ groups with 800+ contacts.

So we’ve got** 640000+(more than 800*800) rows in database table"ofRoster", 80000+**(more than 800*100) rows in database table “ofGroupProp” .

When I shared all 100+ groups for all users, only 120 users can log on.

Then I have to reinstall openfire, Just shared 4 public dept group to all users , now 300+ users can log on.

We need to show all dept groups to everyone, please help.

Openfire is a great real time collaboration solution, thanks a lot for all your support and help.

I have been shared just one group in AD via Openfire, that OU is the USER. We put under the sub groups and the useres. Openfire can mapping all users.

We using openfire groups, so I created or users created the chat groups (under the conference.servername).

If its not working, try to modyfy the LDAP connection and the server java settings.

Another hard idea: Try to install the Openfire server to the Domain controller… I know its not good but… maybe will help you.

Thanks for your advice!

I installed Openfire on RHEL6 server, and just as you said it’s hard for me to install Openfire on DC.

You shared just one group with all users? So everyone have a whole contact list with all other users?

For us, that should decrease “ofGroupProp” to 800+, but maybe stay the number of “ofRoster” at 640000+

Chat groups is also a good idea!

We will try to use this before a solution is found.