Maximum concurrent users possible with Smack?

Smack Smack Dev
1

Hello,

I’m trying to stress test a XMPP server using smack. Two questions:

I need to have more than 20k concurrent users on the server but my testing infra-structure only goes up to 4k before it stops logging in the users. I was monitoring memory usage and noticed that in this case the memory usage was a little over 80% so I’m guessing smack might be using too much memory.

Every time I need to login with a new user what I do is:

ConnectionConfiguration config = new ConnectionConfiguration(SERVER);

this.connection = new XMPPTCPConnection(config);

this.connection.login(this.number, this.password);

Presence presence = new Presence(Presence.Type.available);

this.connection.sendPacket(presence);

this.chatManager = ChatManager.getInstanceFor(connection);

this.chatManager.addChatListener(//Do nothing…);

All of these ‘users’ are stored in a Java Map. I can’t discard them because I need these users to be concurrent users so if I delete them the connection will be closed and the user disconnected. My worry is that this might be using too much memory (not sure yet but I decided to tackle this first). My questions are, is there any way to keep a user logged-in using smack without having to use all this overhead? Has anyone been able to get concurrent users to a really high number (e.g. 20k+)? Is it possible to optimize the memory usage in any way here?

One more unrelated question, has anyone gotten this error:

org.jivesoftware.smack.XMPPConnection callConnectionClosedOnErrorListener

WARNING: Connection closed with error

stream:error (conflict) text: Replaced by new connection

at org.jivesoftware.smack.tcp.PacketReader.parsePackets(PacketReader.java:226)

at org.jivesoftware.smack.tcp.PacketReader.access$000(PacketReader.java:47)

at org.jivesoftware.smack.tcp.PacketReader$1.run(PacketReader.java:81)

I realize this happens because I want to re-login with a subscriber without making him log-out (i.e. I don’t .disconnect() before calling .login() again on the same subscriber - this is to simulate a network outage). The problem is that this exception is cluttering my output so I have to redirect stderr. I haven’t been able to override the exception as well. Anyone have any ideas on how to stop this exception?

2

I too am a new user to the smack. I signed up for this community to ask exactly the same question.

I’ve been able to go up to 10K connections on Linux machine. This results in hitting the per-process thread limit of 32K.

For each new connection, it creates 3 threads viz: Smack Listener Processor , Smack Packet Reader , Smack Packet Writer.

For me, this is the limiting factor and I want to get rid of these/any workarounds.

3

Answering to the connection error that you are getting, you may have to add ConnectionListener to the XMPP connection, overriding connectionClosedOnError method to handle exception appropriately.

4

Hi Gautham,

You’ve gotten to 10K users which is a lot more than I did, how did you set up your program?

Currently, I have a ‘Subscriber’ class which contains the smack-related objects, namely XMPPTCPConnection and chat listener. I’m using multi-threading because I need this test tool to be fast. What I do is define some arbitrary number of threads (I usually tried with 20~30 as a starting point) and divide the number of subscribers between them (so for example, if I were using 1000 subscribers and 20 threads then each thread would handle 50 subscribers), then each thread would have a for loop which simply calls .connect and .login for each of the subscriber objects. As I said before, whenever I get close to 4K subscribers logged in my tool seems to stop all execution and no more subscribers get logged. From my initial profiling I saw that the memory usage on the machine was up to 80% and I saw that the smack api was using most of it (I’m not used to doing profiling in Java though so this might not be correct).

I know that my XMPP server is not to blame here because I’ve used different machines at the same time and have gotten up to 25K subscribers - I want to try and get this number with one machine if possible. All my machines are Linux as well.

Do you have a similar setup? How did you see you were hitting the thread limit (perhaps I can do the same thing here and find out that it’s the case with me as well)?

5

Hi Jesse,

I have lin machine with 16 GB ram. I allocated 10GB for the java client process. I also set concurrent mark sweep GC. With all this, I could see that the heap memory utilization of java process was about 1GB for 1000 connections. However the threads spawned by the process was 32K, which is the limit.

My program had single threaded execution calling .connect and .login methods. But it didnt take much time for 10k connections.

You may try connecting to your java process through jconsole and see the mem utilization, CPU util, threads spawned etc.

You can take thread dump and see exactly how many threads of what type are spawned.

jstack -l

You can also take histogram of the heap for further analysis.

jmap -histo:live

You can try adjusting heap sizes (-Xms , -Xmx values)

Below are the options that I used for running my program:

-XX:+UseBiasedLocking -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:ParallelCMSThreads=2 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9192 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xms10g -Xmx10g

6

Smack was not designed to create 10.000s of connections within the same JVM (of course having multiple XMPPConnection within the same JVM is fine). I would recommend using a more lightweight XMPP library written in C or in Lua for stress testing.

You could also use multiple JVM processes, but still, the overhead caused by the threads Smack uses per connection make it not the ideal library for stress testing.

7

Gautham,

Have you tried changing your Linux stack sizes and/or number of threads? (as per: multithreading - Maximum number of threads per process in Linux? - Stack Overflow )

I was using 2GB of memory before, I’ve just upgraded to 8GB but I’m still only getting up to 4K subscribers, I even tried changing the variables above. I’m really not sure what is going on.

8

4K or 4096 sounds like it could be the maximum file description ulimit for a single process, which is preventing you from creating more TCP connections.

9

Yes! Indeed this was the problem, thank you very much Flow!

The limit of file descriptors was set to 1024 in my machine, I set it to 1024000 and now I’m getting to 10K subscribers as well.

Just for reference, to set the file descriptor limit you use 'ulimit -n ’ in case anyone wanders here.

Use the following command: ‘ps huH p | wc -l’ I’ve found out that I’m hitting 32K threads as well, I tried tweaking some of the numbers, like ‘ulimit -s’ for stack size and /proc/sys/kernel/threads-max but so far 10K seems to be the maximum. I’ll respond to this thread again if I find any way to go over this limit.

10

Jesse,

You better add any change to file-descriptors limit / stacksize or any other resource limit in /etc/security/limits.conf or in your startup script.

Changes made using ulimit command is applicable only to the current session.

11

Gautham,

Yeah, thanks for that, better than having to reset those numbers every time.

Did you have to tweak any of the numbers I mentioned to get to 10K? If not, then have you tried tweaking them to see if you get over 10K subscribers?

I’ve had no luck so far. Even when I get to 10K subscribers the maximum physical memory that is used is 5.5GB with 35GB of virtual memory being used (I measured using top and pidstat), so memory doesn’t seem to be the issue at this point anymore but there is still something limiting the number of subscribers possible even though I’ve tweaked the number of threads, file descriptors etc…

12

Hey, just an update. I was able to go up to 21k concurrent subscribers after tweaking these:

  • /proc/sys/kernel/pid_max
  • /proc/sys/vm/max_map_count
  • /proc/sys/kernel/threads-max
  • ulimit –n
  • ulimit –s

I also upgraded the machine to 16GB. Resource usage on my machine was 74.1g virtual memory, 10g memory , 70% memory use.