Load generators

Archive Project Pampero
1

It would make sense that if we’‘re building a massive scalable server, we should throw as much load at it as possible to see what it can do. It’‘ll also give an ability to benchmark changes to make sure that we don’'t regress performance wise for any changes that get introduced.

However, what does a XMPP generator look like. Two projects have been mentioned as candidates. One is OpenSTA (http://www.opensta.org), which I believe was used to load test messenger, and Grinder (http://grinder.sf.net). What have you used that has been successful?

Regardless, of what technology is used to generate the load, we’'ll need a set of use cases to help give us simulate typical users.

The next piece of the puzzle is actually generating 100k connections to Pampero. If JiveSoftware.com is happy to provide the equipment to do such a test, then we’‘re in good shape. We might be able to find a split. The server is running somewhere and all contributors and volunteers use their development machine to throw load at it. If we really want to see the limit we could rent a cluster (Sun has their $1/cpu-hour deal that we could take advantage of. For $100 you can get 100 CPUs for one hour. Maybe they’'d even donate the resources to such a worthy cause since they announced that XMPP as the their standard for their version of EIS http://www.sun.com/smi/Press/sunflash/2005-03/sunflash.20050330.2.html). We could have a shoot out.

Noah

2

Well,

I think find some people that have hardware isn’'t dificult.

From me, I have 3 computers on a 10Mb connection to internet.

Then, my boss have some machines and can provide them,

and then The society which I work for can provide a server, perhaps 2 on 100Mb connection,

one with 2.4GHz, another with Bi Xeon and 2Mb of RAM

3

Arg, I forget always something !!!

So, for the 100 000 Connection, for me they are 100k connections to the main server not to one connection manager, perhaps i’'m mistaking…

so We can think it will can have at least 20 k connection by connection manager, and I think we can go to 50 k… another stuff which will be decisive, its the load needed by the main server to treat all users…

I don’'t know really how much need JiveMessenger, but for 100 k connections, perhaps it need something not negligible.

4

Thanks for the donation. I forgot to mention the third option…the kindness of strangers/community.

To generate 100k we might need to find bigger pipes than 10mb and 100mb. Although the 10mb will give us a sense of what a server behind a T1 would like (1.5mb).

5

Arf scuse, i’'m confused

I’'m a little tired and say bad stuff, the 10Mb is on download, I have 1Mb in upload.

Well, but the 100Mb keep 100Mb so 10mb in upload.

The only problem with our servers on 100Mb are the hours of tests, the society which i work for is in France(GMT +100), and we have some clients on the server, so the use of test must be here the night and not the friday night ( here )

6

Here is another:

http://www.slamd.com/

7

But I don’'t really understand how it works ?

Doesn’'t we need something which connect really as jabber user ?

to test the both servers ?

8

You can write a custom script that connects as a jabber user, just sending XML over TCP.

9

ok, so this libs are only to execute a lot of times the same script ?

10

Yeah, something like that. I don’'t know for sure as I just found it today but it looks pretty so I think at least that makes it worth investigating

11

Hey Noah,

Other options that we might want to use are url=http://unix.freshmeat.net/projects/idx-tsunami/IDX-Tsunami[/url], url=http://sourceforge.net/projects/jabbertestJabber Test Suite[/url] and url=http://www.coversant.net/downloads/XMPPStressInstall.msiSoapBox XMPP Stress Test Tool[/url]. BTW, we didn’‘t use OpenSTA to test JM scalability and I’'m not sure if we may use it for XMPP.

Regards,

– Gato

12

Hello all,

Let me say something as I have some experience with such issues.

I work on alternative XMPP server implementation in Java and have recently worked

on very similar issues.

I have also found many scalability issues in my implementation and I am now working

on solving them.

Solving them means also testing which generates other issues.

First of all you don’'t need huge network connection to do all these tests.

You actually don’'t even need to generate a lot of traffic. At least initially.

Please note there are at least 2 separate parts to test:

  1. Connection manager

  2. Session manager

Assume: connection manager forwards all user requests through one

connection which can be kind of JEP-114 implementation to session manager.

So you could easily test session manager with only one connection open.

Over this one connection you can simulate many concurrent open sessions.

Theoretically any number: 1k, 10k, 100k, 1M.

You can generate as heavy traffic as you want and this doesn’'t require

large network to perform tests. It could be even done through localhost.

Connection manager:

This is kind of proxy. The main duty of this proxy is to perform XMPP stream

initializations, allow user to use SSL or TLS and maybe process some basic requests.

The main issue for this component is how it can server many, many concurrent connections. And how heavy is load on each of these connection is not THAT important.

Additionally note: it is not easy (if actually possible) to open more than 1k connections on MS Win system. Default configuration doesn’'t support more than 1k open TCP/IP ports. There are recipes on the net how to allow more. Neither of them worked for me.

But I doubt it is possible to open more than 65k on one windows box.

It is much better on Linux. I could easily configure it to allow about 10k concurrent TCP/IP connections. Could force Linux to reliably open more than 10k. If there is over

10k concurrent TCP/IP connections Linux kernel takes over 50% of CPU time. Connections are very unreliable, many dropped, errors and so on…

It looks quite well for up to 1k connections though.

And as for windows, reading Linux documentation I have got impression that is it not

possible to have more than 65k concurrent connections.

Here are some results from my tests:

http://tigase.dev.java.net/performance-tests.html

These tests are for my server used my own testing framework.

Artur Hefczyc

13

Artur, thank you for your valuable insight. I was not aware of the limitations of windows and after you post I discovered that XPsp2 will limit the number connections to 100! I’'m sure that this can be overridden. Do you know if this limit applies to their Server or DataCenter editions?

You hit the nail on the head with regards to our current approach to this task. The purpose of the connection manager is to aggregate multiple inbound connection into one connection back to the server. Our current desire is 10k per CM and 10 connections to Messenger itself. The CM’'s will handle all connection related salability issues and Messenger will have to be very fast at handling such a large volume.

One thing that we haven’‘t yet determined is how much load will be pushed through at any given time. For isn’‘t, handling 10k connection is easy (well, pending the winxp limit) if you don’‘t actually have to do anything. 10k connections idling away isn’'t a good test assuming that your connection handling code is capable of such numbers.

The next piece of the puzzle is figuring out what kind of test profiles we want to provide. Of the top of my head, I think there are idlers[/b] (this would actually be interesting in terms of presense updates as connections come and go). Bursty[/b] chatters, chatters that talk in small bursts and then idle for most of the time. Long winded[/b] chatters. Bots[/b] and then special[/b] uses cases like broadcast from admin. I’'d love to get some profiles from a live jabber server that I can scrub out the content and replace with nonsense, but at least keep the chat style.

Noah

14

Hm, I wasn’'t aware of last WinXP SP2 limitation. Most of my tests I do on Gentoo Linux, and Win2k. My resources are a bit limited.

I am sure (hm this is just logical guess) Windows Servers should be able to handle much, much more concurrent connection. They are just used as heavy load servers somewhere.

I have forgotten to mention about something. I experienced some issues with handling 10k concurrent connections on Linux and 1k connections on Windows. But tbh my tests were somehow unrealistic. I do some extreme tests on my server to know exact limitations.

For example I do XMPP session binding (after socket connection, XMPP stream init, TLS, SASL authorization) and disconnecting from the server 1000 times in row

or connect to the server register new user account, authorise and delete user account

or connect to the server send 100 messages to existing users and disconnect.

As quickly as possible.

All these tests are performed with different number of concurrent connections.

These are very unrealistic tests but in such tests we can find limitations of our software.

I agree that opening 10k (or even 1M) iddle connections is not a good test.

Good tests should answer certain questions like:

  1. How application behaves in common case

  2. What are application limits

  3. How application behaves in extreme cases

As I mentioned above to test my server I implemented also dedicated test framework.

It is designed specificly for testing XMPP servers. It contains built-in scripting language

for creating different test scenarios.

Using this I have implemented following 3 test scenarios:

  1. Functional tests: http://tigase.dev.java.net/functional-tests.html

    script for this is here: http://tigase.dev.java.net/source/browse/tigase/scripts/all-xmpp-tests.xmpt?view =markup

  2. Performance tests: http://tigase.dev.java.net/performance-tests.html

    script for this is here: http://tigase.dev.java.net/source/browse/tigase/scripts/perform-xmpp-tests.xmpt? view=markup

  3. Stability tests: http://tigase.dev.java.net/stability-tests.html

with script available here:

http://tigase.dev.java.net/source/browse/tigase/scripts/stability-xmpp-tests.xmp t?view=markup

Scripting language I have implemented should be flexible enough to use it for

implementing other scenarios as well.

Well, what for I am writing this all in here?

I wonder how could you cooperate together to benefit both of our projects.

If you have any ideas they all are very welcome.

The first step in my opinion would be publishing kind of JEP-114 which could

be used for communication between CM and SM.

As next step (or just concurrently) could be implementation of CM although

it is almost implemented in my server but we could create kind of external component

independent of both Jive or Tigase server.

What do you think?

Artur Hefczyc

15

Here are a couple of links about the new connection restriction that appeared in WinXP SP2. Those links explain the “problem” and provide a patch. I applied that patch many many months ago and since then I was able to have more than 10 connections.

Links:

http://www.phex.kouk.de/mambo/content/view/49/33/

Regards,

– Gato

16

Great! I’‘m glad your interested in working on this project. Since you’'ve been deep in development on your project and are willing to share, your input is very valuable to us.

I’‘m actively working on a few support pieces to the puzzle since I seem to have the expertise and the time to donate. I’‘m currently working on instrumenting messenger with JMX so I can actively profile what’‘s going on in each component as well as tweak parameters at runtime. This plugin is called meter and is in the pompero branch in svn. The next piece of the puzzle is the test harness. This is up in the air and our last Wednesday chat (which Matt will post shortly) ask for volunteers. So far no one has come forward to take responsibility so it’‘ll most likely fall in my lap. If you’‘re willing to help and it sounds like you’‘ve done quite a bit of the ground work already, then we should join forces and come up with a test environment that suits our needs. It’‘ll requires micro benchmarking, which is what you’'ve done already with the full throttle approach, and functional interaction which is what I discussed in my last post.

Organizationally, we’‘ll need to decide where the source code lives. Jive has some requirements on who can contribute (although they’‘re minimal) and hosts svn. Obviously, we’‘d like everything we’‘re working on under one “roof” but it’'s not a hard requirement in my book.

Once we get the test harness and the metering put together, it’‘s time to jump into the JEP protocol. There’‘s already been some discussion, but nothing ratified and it’'s all in the air as of now. I think that building a prototype and then standardizing, while controversial, will get us to the end point as soon as possible. So using JEP-0114 as a references, I think the next step is start looking at CM and Messenger in regards to scalability.

At this point, we’'re at a cross road. Do you want to remain independent of jive messenger or do you see benefit in combining the two projects? For the most part, CM will be part of this project because of the interest in making messenger and CM as closely integrated as possible.

17

I was thinking about merging projects JiveMessanger with Tigase. Well that is not that easy, both of these projects are created differently.

For example it is all NIO based including SSL and TLS, uses own XML parser implementation designed specifically for XMPP streams and so on…

At least some goals are different.

I don’'t want to give up my project. On the other side I work on it alone in my spare time. So I have very limited resources in terms of time and HW. I have however a lot of ideas some of them are partially implemented.

I still think there is a way to cooperate with benefits for both projects.

So my approach is to keep source codes and whole projects separately where it does make sense. Especially CM in tigase is already implemented. At the moment it can’'t be run separately but I am reimplementing certain parts of my server to make it possible to run separately any number of CM instances connecting to SM.

So it will be implemented regardless you have your own. But still we can benefit from having 2 implementations. You could test your server with mine CM and I could test my server with yours.

As I mentioned before I created test suite for XMPP servers. It is not published yet.

I can publish this suite with some sample test scripts but I don’'t have time to create more test scripts covering different scenarios. So if you have resources to work on scripts and share them with me I would be happy to correct all bugs and improve suite for our needs.

If your tests scripts are publicly available in your source code repository I am OK with that.

From my point of view working together on JEP is definitely the first step.

What happens next we will see. As you mentioned we can be at a cross road, yes that is possible. But there are also other possibilities.

Artur Hefczyc

18

Totally understand your reasoning. I just wanted to throw it out there to see if you interested. JS.org has more resources, but we’‘re all spread out. We’'re not a project like Tomcat or Struts…yet ;).

As far as test scripts are concerned, I think we’‘d have to take a look at your framework and see it meets what we’‘re looking for. If we can morph that into what both suits are needs, then we’'re in good shape. As for generating scripts, there might be some individuals that can contribute on our side…at least agree to let us audit their conversations and use their chat patterns as input to the test scripts.

JEP-0114X…let’'s take this to another thread…http://www.jivesoftware.org/community/thread.jspa?threadID=15956

Noah

19

Hey Artur,

It is exciting to see you around. I think that project Pampero is a good opportunity for joining efforts that will improve both servers. I agree with you that the first step would be to come up with a new JEP (probably based on JEP-114).

Once we have a JEP the next step would be to start implement it. In your case you may want to modify the CM implementation in Tigase to follow the JEP and/or help us implement the Pampero’‘s CM. Since you will be able to use Pampero’‘s CM with Tigase there exists the option to replace Tigase’‘s CM with Pampero’'s CM.

Anyway, depending on you spare time, you may help us by sharing your experience, knowledge or implementing Pampero’'s CM.

Regards,

– Gato

20

Anybode use SoapBox XMPP Stress Test Tool, IDX-Tsunami for testing wildfire?

Kobit please say can i use your test utility for test Wildfire server?