Need Help Tuning Messenger for Large Implementation

hughesmk · August 18, 2005, 12:44am

We are using Jive Messenger with a custom client. End state target is 160K+ devices connecting concurrently across a distributed infrastructure.

We have a small pilot going currently with about 2K endpoints running against a single server and unfortunately we are struggling to get Jive Messenger configured well enough to stay running. I have seen reference in the forums that a properly sized server should be able to handle 5K or more endpoints, even at that, this would require 32 servers plus failovers and databases, we use Sametime from IBM on half that number of servers including the backups, so tuning and balancing is going to be pivotal if we are to succeeding with using Jabber.

The server is a 8 way machine with 8GB of RAM running Windows 2000 Advanced Server. We are using SQL Server 2000 with Jive Messenger.

We have created the messengerd-service.exe.vmoptions with the following options which all appear to be ignored

-Xms512m

-Xmx1024m

-Xss128k

The server runs out of memory after only 350 clients.

If we start it from a batch file using the following command it is good to about 1700 clients

start “Jive Messenger” “C:\Program Files\Jive Messenger\jre\bin\java” -Xms512m -Xmx1024m -Xss128k -server -jar …\lib\startup.jar

Unfortunately nothing seems to be able to get it past that limit.

We have tried the release version as well as the daily builds. We have ensured that we are using the server vm. We have removed all plugins, and turned off everything that we can think of including the admin ssl port, just to try and reduce the memory enough to get beyond the 1700 client limitation.

The specific error that we are getting is:

18:46:26.411 WARN!! [Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=9090]] org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:632) >01> EXCEPTION

java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method)

at java.lang.Thread.start(Unknown Source)

at org.mortbay.util.ThreadPool$PoolThread.enterPool(ThreadPool.java:448)

at org.mortbay.util.Pool.newPondLife(Pool.java:368)

at org.mortbay.util.Pool.get(Pool.java:304)

at org.mortbay.util.ThreadPool.run(ThreadPool.java:368)

at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:616)

Any assistance in getting messenger configured to handle the load would be greatly appreciated.

Noah_Campbell · August 18, 2005, 3:15am

You’'re definitely going to push the limits of messenger (or any server for that matter) with your sizing requirements.

Look here http://blogs.sun.com/roller/page/moazam?anchor=why_can_t_i_allocate for information on how to get more connections. This should get you more connections per machine. Going by your stats, if you double your memory (which is about all you can do on a 32-bit jvm on windows) you’‘ll get 600 clients per jvm. Since that’‘s using only 1/4 of the memory, you’‘ll probably be able to get another 1200 (2 more jvms), before the machine starts to bog down with the thread switching overhead. You’'ll need to create 3 total jvm per machine and bind them to a unique network interface and load balance across your server farm.

The other option is find a 64-bit cpu and os. Solaris 10 and an amd come to mind.

Look at http://www.spec.org/osg/jAppServer2002/results/res2003q3/jAppServer2002-20030722 -00013.html for an example configuration similar to your requirements. If you have 160k devices and IBM with sametime already installed, then you won’‘t be shy about purchasing hardware that’'ll get the job done.

What’'s your timeframe for delivery?

Cheers,

Noah

matt · August 18, 2005, 4:11am

To add to Noah’‘s post, a key way to increase scaling is a two tier architecture. Basically, multiple slave instances aggregate connections to a single core router. I helped a guy put together a Google Summer of Code proposal to work on this (but it was rejected). I still the the work would be very interesting – probably about a two month project (one month dev, one month testing). We’'re definitely looking for people to sponsor this work.

-Matt

Summer of Code Proposal: Massive Scaling for the XMPP Instant

Messaging Protocol

Open Source implementations of the XMPP (Jabber) protocol such as

jabberd are widely and successfully deployed.

However, most Open Source implementations cannot be deployed for

very large groups of users (more than 10,000 concurrent

users) because a single server cannot maintain a massive number of

client TCP/IP connections. I propose creating an XMPP protocol

extension and implementation that will let multiple CMs (Connections

Managers) aggregate many client connections to a single Router. I

will deploy my implementation of the protocol extension in a test

environment to demonstrate handling 100,000 simultaneous client

connections with a single Router (assuming an arbitrary number of

CMs).

XMPP components are typically implemented using the JEP-0114

protocol extension. However, this protocol is quite dated and the

community seems to be in general agreement that it should be

updated. My proposal is to define a more modern component protocol

(superseding JEP-0114, probably using TLS and SASL as defined in RFC

and at the same time create a profile that will allow the

Router-CMs routing protocol described above. This fits in well with

the standard Jabber architecture while providing added value to the

community beyond just improved scalability of server

implementations.

My deliverables for the project will be as follows:

New component JEP which includes the Router/CMs routing feature

for client and server-to-server connections.

Implementation of a CM. This will be written in Java under the

Apache license and will leverage the existing Open Source

Whack component library (Apache license).

Modifications to the Jive Messenger Open Source XMPP server to

allow slave server connections.

Test environment that demonstrates scaling to 100,000 simultaneous

client connections.

matt · August 18, 2005, 4:12am

As another tip to help scaling, you might try dialing back the stack size even a bit further. We haven’'t tested how low it can go, but you might try 64k for example.

-Matt

hughesmk · August 18, 2005, 5:16pm

All good information, and we will take it all into consideration. At this point however, we still cannot get Jive Messenger running as a service to handle more than 350 clients, only when run from a batch file as shown below does it scale beyond that point.

matt · August 18, 2005, 5:22pm

Oh, I missed the fact that you were having problems making the serivce work. You can pass in params to the service by using a “vmoptions” file. A copy and paste from the install guide is below.

Regards,

Matt

Custom Parameters

Advanced users may wish to pass in parameters to the Java virtual machine (VM) to customize the runtime environment of Jive Messenger. You can do this by creating vmoptions files in the bin/ directory of your Jive Messenger installation. For the Windows service, you’‘d create a new text file called messenger-service.exe.vmoptions. Each parameter to the VM should be on a new line of the file. For example, to set the minimum and max VM heap size to 256 MB, you’'d use:

-Xms256m

-Xmx256m

To create parameters for the normal launcher, create a file called messengerd.exe.vmoptions (since the messenger.exe launcher invokes the messengerd.exe executable to actually start the server).

jimmy · September 15, 2005, 4:41am

Hi,

I saw your post in the jive forun in Aug 17, 2005 7:44 PM .

Seems you can create 1700 clients by using the following command.

you said:

{

If we start it from a batch file using the following command it is good to about 1700 clients

start “Jive Messenger” “C:\Program Files\Jive Messenger\jre\bin\java” -Xms512m -Xmx1024m -Xss128k -server -jar …\lib\startup.jar

}

My directory is just exactly the same as this command. and i also sure the mentioned files (e.g. startup.jar) exist.

to start the jive messenger server. I type

{

start “Jive Messenger” “C:\Program Files\Jive Messenger\jre\bin\java” -Xms512m -Xmx1024m -Xss128k -server -jar …\lib\startup.jar

}

in the DOS mode.

But seems nothing happen (the server didn’'t launched) after I type exactly the same command in my DOS mode.(I just copy it exactly).

My question are:

Is this exactly one line command?

or 2 line?

e.g. firstly type:

start “Jive Messenger”, then press enter.

secondly type :

“C:\Program Files\Jive Messenger\jre\bin\java” -Xms512m -Xmx1024m -Xss128k -server -jar …\lib\startup.jar

thne press enter

is something like “” needed.

e.g. should be “Jive Messenger”, but not Jive Messenger?

should be “C:\Program Files\Jive Messenger\jre\bin\java”, but not

C:\Program Files\Jive Messenger\jre\bin\java?

Thanks a lot

Jimmy

Noah_Campbell · September 15, 2005, 1:11pm

I use a different JAVA_HOME than what was given by messenger (probably because I’'m getting it directoy from svn. Regardless, here are a few thinks to try.

Remove the -server
Remove the start “Jive Messenger”

This way output will be directed to your immediate console. The start command is meant to open a new console or application and run it. If there is a failure (like -server not being recognized as an option) then a screen will flash (the start command starting and exiting immediately afterwards).

HTH,

Noah

jimmy · September 16, 2005, 3:06am

Hi,

Thanks for reply.

You said

{

Remove the -server
Remove the start “Jive Messenger”

}

So after type “jre\bin\java -Xms512m -Xmx1024m -Xss128k -jar lib\startup.jar”

in dos mode under “C:\Program Files\Jive Messenger” directory.

The Jive messenger server would be launched?

But I have the following exception occurs.and Jive messenger server didn’'t launched.

Am I misunderstood what you want to said here?

p.s.

Originally I type

{

start “Jive Messenger” “C:\Program Files\Jive Messenger\jre\bin\java” -Xms512m -Xmx1024m -Xss128k -server -jar …\lib\startup.jar

}

But nothing happed.

Thanks a lot,

Jimmy

C:\Program Files\Jive Messenger>jre\bin\java -Xms512m -Xmx1024m -Xss128k -jar lib\startup.jar

java.lang.ClassNotFoundException: org.jivesoftware.messenger.XMPPServer

at java.net.URLClassLoader$1.run(Unknown Source)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(Unknown Source)

at java.lang.ClassLoader.loadClass(Unknown Source)

at org.jivesoftware.messenger.starter.ServerStarter.start(ServerStarter.

java:80)

at org.jivesoftware.messenger.starter.ServerStarter.main(ServerStarter.j

ava:46)

Noah_Campbell · September 16, 2005, 3:24am

I would of thought that starting the server from c:\Program Files\Jive Messenger[/i] would have worked, but alas, it doesn’'t.

However, you change to the bin directory and execute the following command:

…\jre\bin\java -Xms512m -Xmx1024m -Xss128k -jar …\lib\startup.jar

/code

That should do the trick.

Noah

Jean_Vence · September 27, 2005, 3:27am

What kind (amount) of sponsorship are you looking for?

matt · September 27, 2005, 5:37am

What kind (amount) of sponsorship are you looking for?

Having engineering time sponsored would justify much faster work on Project Pampero. Please drop me an email if you’'d like to discuss further.

Regards,

Matt

Michael_J_Long · September 28, 2005, 7:21pm

I have noticed that the vmoptions does not operate as described. I have confirmed this by trying a variety of options with peculiar results.

In my particular situation, I am stress testing (using The Grinder) a Jive server (Win2K) with 1000 simultaneous logins. Using the messengerd.exe with the default VM settings (i.e., no vmoptions), I received an OutOfMemoryError with a “Java Heap Space” after about 860 connections. I modified vmoptions to add -Xmx (from 128m to 512m) and -Xss128k but received OutOfMemoryError with a “unable to create native thread” after making fewer connections than with the default VM settings (more on this later).

Based on other posts in this thread, I decided to run the server from the command line by using java.exe (using startup.jar) with -Xmx128m and -Xss128k. This seemed to work just fine, I was able to make 1000 concurrent connections without a problem. I can actually run with as little as 80m (-Xmx80m) and still acheive my goal.

As I found my results with vmoptions to be very strange (the higher the max., -Xmx, the fewer concurrent connections I was able to make). Also, because I was able to reach my goal using the startup.jar, I figured the way the server is started through messengerd.exe has problems. I looked at the *nix scripts for starting the Jive server and found the following:

nohup “$app_java_home/bin/java” -server -Dinstall4j.jvmDir="$app_java_home" -Dinstall4j.appDir="$app_home" -Dexe4j.moduleName="$prg_dir/$prg" $INSTALL4J_ADD_VM_PARAMS -classpath “$local_classpath” com.install4j.runtime.Launcher start org.jivesoftware.messenger.starter.ServerStarter false false “$prg_dir/…/logs/stderror.log” “$prg_dir/…/logs/stdoutt.log” true true false “” true true 0 0 “” 20 20 “Arial” “0,0,0” 8 500 “version 2.2.0” 20 40 “Arial” “0,0,0” 8 500 -1 -DmessengerHome=$app_home -Dmessenger.lib.dir=$app_home/lib “$@” &[/i]

I adapted that command to use it on my Win2K box and received the same errors as the messengerd.exe with the default VM options, which is not unexpected, I didn’'t specify any additional VM options to increase the amount of memory. After adding “-Xmx128m -Xss128k” in the adapted command, I was able to create the same concurrent 1000 connections.

As our production system will be *nix, I know that I can specify the VM options in the INSTALL4J_ADD_VM_PARAMS and acheive the goals I have set.

However, I would like to know why the vmoptions is not working as advertised. Does messengerd.exe use a different command than that specified in the *nix script?

I suspect that this is an Install4J issue, not a Jive issue but I thought you should be made aware of it.