Performance issues with MUC?

Hi,

I’'ve spent the day testing the performance on 3.2 wildfire binary and RHEL 4. My testing includes 126 chatrooms with a history setting of 60 stanzas with logging in PostgreSQL.

So I attempt to connect various numbers of simulated users to these rooms over localhost connects in an unthrottled way. It seems wildfire can handle 2 or 4 users before it starts bombing out (wildfire starts pegging CPU) and dumping out messages like:

2007.02.12 01:16:27 org.jivesoftware.wildfire.nio.ConnectionHandler.exceptionCaught(ConnectionHandle r.java:104)

java.lang.OutOfMemoryError: Direct buffer memory

at java.nio.Bits.reserveMemory(Unknown Source)

at java.nio.DirectByteBuffer.(Unknown Source)

at java.nio.ByteBuffer.allocateDirect(Unknown Source)

at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)

at sun.nio.ch.IOUtil.read(Unknown Source)

at sun.nio.ch.SocketChannelImpl.read(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.j ava:232)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcesso r.java:206)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$500(SocketIoProce ssor.java:44)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:502)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:43)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Something doesn’'t seem right. Any ideas? What sort of performance should I expect with MUC?

Perhaps my testing client is naughty? Is there a benchmark that test MUC rooms as well?

thanks,

daryl

Good morning,

I thought that maybe MUC history was to blame, so I turned off MUC history on login (which required server reset) and tried again. I still get the same behaviour, pegged CPU for 5-10 minutes with only 1 user with 10 resources connecting to 126 rooms.

Any ideas?

daryl

“java.lang.OutOfMemoryError: Direct buffer memory”

There is your issue. Checked the allocated memory you are giving the server, then check on how much memory that Wildfire is using.

Jeff

Thanks for the help. Well, I saw your recent post about stack size and am trying this currently:

-XX:+UseParallelGC -Xss128k

I can’'t access the admin console when the CPU is pegged, but ps reports

jabber 12263 78.7 57.6 924660 597164 pts/1 Sl 17:24 4:56 /opt/wildfire/jre/bin/java

hmmmm,

daryl

How are you starting your server? have you done anything else with your wildfire script besides the -XX entry above? what is the total sys memory you have installed?

Also, is there anyway to simulate your connections off server, and how are they authenticating?

Jeff

How are you starting your server?

cd /opt/wildfire && ./bin/wildfire start

have you done anything else with your wildfire script besides the -XX entry above?

nope.

what is the total sys memory you have installed?

1 GB physical, 2 GB swap

Also, is there anyway to simulate your connections off server,

I’'m not sure what you mean by off server. I am in the testing phase now and all connections are simulated. I have a simple Twisted python script that logs 1 user in under 10 different resources and then connects those 10 resources to 126 chatrooms.

and how are they authenticating?

Postgres.

thanks!

daryl

You aren;t using Active Directory, was confused initially…

INSTALL4J_ADD_VM_PARAMS= “-Xms128m -Xmx512m -XX:+UseParallelGC -Xss128k (or 256k if you like)”

the -XX:UseParrallelGC is supposed to be for MutliProcessor systems only. (Java is ALSO supposed to do it for you, but it did help my performance so it stays in my start script)

“Simulate off server”, I mean is there anyway to use your script from a second box and point it at your WiFi box to do the connections. running the simulations might be sucking up some resources that you might need.

What does “top” say on the server as far as resources used, and by what procs…

Jeff

Message was edited by: jeff_garner

Message was edited by: jeff_garner

INSTALL4J_ADD_VM_PARAMS= -“Xms128m -Xmx512m -XX:+UseParallelGC -Xss-128k (or 256k if you like)”

Are you sure you have your dashes right? I have now set:

INSTALL4J_ADD_VM_PARAMS=“-Xms128m -Xmx512m -Xss128k”

I have a Pentium D and am currently running with a UP kernel.

“Simulate off server”, I mean is there anyway to use your script from a second box and point it at your WiFi box to > do the connections. running the simulations might be sucking up some resources that you might need.

Good thought, I’'ve been testing over both localhost and remote connections. The behaviour is the same.

What does “top” say on the server as far as resources used, and by what procs…

1.7 GB VIRT 909MB RES

ps auxww | grep java

/opt/wildfire/jre/bin/java -server -Dinstall4j.jvmDir=/opt/wildfire/jre -Dexe4j.moduleName=/opt/wildfire/bin/wildfire -Xms128m -Xmx512m -Xss128k -classpath

My current test is to connect 1000 resources of 1 user with no MUC interaction. Pegs CPU indefinetely. ufff.

thanks for your continued help. I must be doing something silly.

You are correct, I do not have my ‘’ - ‘’ correct. and there shouldn;t be one at ‘’-Xss128’’

Have fixed that, so thanks for pointing it out to me. Been a long day already.

what happens if you start off with connecting 1 user with say 256 connections? Let’'s see if you start off small at what point it breaks.

Jeff

I may have been DOSing my server

I modified my client script to throttle the connections and the wildfire server is behaving nicely now. I got 1000 users logged in without memory issues. Time to try MUC in this matter.

thanks so much for your help. I’'ll report back once this works or fails

LOL… well atleast you found the root cause of the issue. Good luck.

All good now. Yippee! Now, if my geotrust SSL cert would only work, we could make this production

thanks!

good to hear! Good luck on the certs. I still have issues with those from time to time…