[Ubuntu] Can not login openfire server for over 1000 accounts

Hi,

I am new in Linux OS, and I setup a server with Ubuntu 10.04 and openfire 3.6.4. I can login this openfire server with pidgin, but server will be no responding after over 900 accounts are logged in. Please give me some idea to solve this problem.

Symptoms:

  1. Server will be no responding after over 900 accounts are logged in.

lsof -p XXXX | wc -l command returns 1078~1085 in many testing cycles (XXXX is ID of Java)

  1. I found following information in log files of openfire (warn.log)

java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConne ctor.java:75)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:475)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:166)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java :124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)

What I did for this issue are as follow:

1.Modify “/etc/default/openfire”

DAEMON_OPTS=“-Xms512m -Xmx2048m -Xss1024k -Xoss1024k -XX:ThreadStackSize=128 -XX:+PrintGCDetails -Xloggc:/tmp/Openfire-gc.log -XX:MaxPermSize=128m -Djava.net.preferIPV4Stack=true -XX:+HeapDumpOnOutOfMemoryError”

2.Modify Ulimit by “ulimit -SHn 4096”

ulimit -SHa
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

3.Modify file-max in /etc/sysctl.conf to 400000 and do "sysctl -p"

4.Add settings in /etc/security/limits.conf (openfire is user name)

  •     -    nofile   4096
    
    openfire soft nofile 4096
    openfire hard nofile 4096

5.To unmark a line in /etc/pam.d/su

session required pam_limits.so

  1. I did all above modifications, but is still failed
  1. After I open over 500 files by Office and gedit and do the login test again, issue remains and lsof -p XXXX | wc -l command also returns 1078.

System configuration:

OS: Ubuntu 10.04(lucid) kernel 2.6.32-25-server Gnome 2.30.2

Java: 1.6.0_18, OpenJDK 64-Bit Server VM (build 16.0-b13, mixed mode)

RAM: 7.8GB

CPU: Intel Xeon E5320 1.86GHz

Attached please find the zip file which includes following data:

1.Warning log of openfire

2.Some settings like limits.conf, /etc/default/openfire …etc

Please give me some idea to solve this problem.

Thanks a lot.
Log and settings.zip (48969 Bytes)

Hi Willy,

please post the ulmit for the user, I wonder whether Openfire runs as “openfire” or as “daemon”

Plase post the output of

su - root
su - openfire
ulimit -a

LG

Hi LG,

I input these three commands in X window terminal, but terminal return nothing.

http://picasaweb.google.com/willy6909/Ubuntu#5526609137476762994

And then I found my system is hang up, I reboot it by press power button.

After system is reboot, it shows “fsck / [385] terminated with status 4”.

How to recover system ?

Thanks a lot for your reply.

Hi LG:

The file system is recovered by a LiveCD and command “sudo fsck -y /dev/sda1”

Sorry for your inconvinient.

Hi Willy,

it looks like “su - root” does not work. Could you do a “ps -ef|grep java” and make sure that the Openfire process runs as user “openfire” and not as user “jive”?

At least the openfired script in the .tar.gz version does use the “jive” user to run Openfire.

LG

Hi LG:

the ps -ef|grep java command will output following information:

openfire@ubuntu:~$ ps -ef|grep java
openfire 1039 1 0 01:07 ? 00:00:13 /usr/lib/jvm/java-6-sun/bin/java -Xms512m -Xmx2048m -Xss1024k -Xoss1024k -XX:ThreadStackSize=128 -XX:+PrintGCDetails -Xloggc:/tmp/Openfire-gc.log -XX:MaxPermSize=128m -Djava.net.preferIPV4Stack=true -XX:+HeapDumpOnOutOfMemoryError -server -DopenfireHome=/usr/share/openfire -Dopenfire.lib.dir=/usr/share/openfire/lib -classpath /usr/share/openfire/lib/startup.jar -jar /usr/share/openfire/lib/startup.jar
tomcat6 1302 1 0 01:07 ? 00:00:09 /usr/lib/jvm/java-6-openjdk/bin/java -Djava.util.logging.config.file=/var/lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -Xmx128M -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp org.apache.catalina.startup.Bootstrap start
openfire 2102 1827 0 06:14 pts/0 00:00:00 grep --color=auto java

I test for user account by reboot Ubuntu system and use account names like jive,ubuntu,…etc to login, but i can only login by the account “openfire” (test by X window login window)

I also update java from openjdk to java-6-jre, but issue still exist.

I have no idea how to run openfire for a specified user.

Can you give some suggestions, thanks ?

Hi Willy,

as you are the user “openfire” and Openfire uses this user you may simply run “ulimit -a” and verify that the ulimit returns 2048 for files.

Is there a reason why you did set “-Xss1024k -Xoss1024”? Do you need such a big stack size?

LG

Dear LG:

“ulimit -a” command will return “open files (-n) 4096” as what i set in limits.conf.

When I first encounter this issue, i thought it is caused by memory, so i search google and add “-Xss1024k -Xoss1024” in openfire setting. If i remove these two settings, issue still exist.

I will try to install Ubuntu system on another disk again and see if this problem still happen.

Thanks.

Hi LG:

Finally, I found the answer and please correct me if I was wrong.

I re-install Ubuntu OS and download sun-6-jre and openfire to install.

After that, I did following procedures:

1.In Terminal, “sudo gedit /etc/security/limits.conf”

Add two lines in this file:

  • soft nofile 65535

  • hard nofile 65535

(* is the user name)

2.In Terminal, “sudo gedit /etc/pam.d/login”

Add one line in this file:

session required pam_limits.so

3.Reboot system

4.After system started, ulimit -a can see open file : 65535

5.In termianl, “sudo /etc/init.d/openfire start”

And now I can login over 3000 accounts into openfire server.

I found this http://community.igniterealtime.org/thread/38543 but it says to add

“session required /lib/security/pam_limits.so” in /etc/pam.d/login.

Which one is correct ?

Dear LG:

I have one more question about configuring “nofile” limit.

Can I set nofile more than 65535 ?

Is there a maximum limitation about this configuration in Ubuntu system ?

Thanks.

Hi Willy,

there is a maximum ulimit, try “grep NR_OPEN /usr/include/linux/fs.h” to get it for your system / kernel. It may be 1048576 (=1024*1024) - I’ll use this value later as a sample value.

Anyhow you may not be able to use it, check the current limit with “sysctl fs.file-max” or “cat /proc/sys/fs/file-max” - it may be 200000. If you want to increase it to a higher or the maximum ulimit then you need to set it with “sysctl -w fs.file-max=1048576”.

LG

Dear LG:

1.Result of grep NR_OPEN /usr/include/linux/fs.h is 1024

2.Default setting of fs.file-max is 819200

If I set “sysctl -w fs.file-max=1048576”, then I still can only login 940 accounts.

Does it mean the system maximum value for file-max is 819200 ?

Thanks.

Hi Willy,

maybe you get an overflow somewhere. Anyway I wonder whether you really want to connect more than 10000 clients.

LG

Hi LG:

I do some tests for file-max.

If file-max is more than 819200, I can only login 940 accounts.

If file-max is 819200, I can login 45900 accounts.

I still need your help to find out the root cause.

Our purpose is to find out the capability of our server machine while it run as an openfire server.

We will test both Windows server and Ubuntu OS.