I ran into an interesting problem today - our server is running normally, except that I can no longer connect to port 9091 with a browser (the SSL-enabled admin console port). Port 9090 (non-SSL) continues to function normally. The only log entry (admin-console.log) that makes any sense seems to be:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at org.mortbay.util.ThreadPool$PoolThread.enterPool(ThreadPool.java:448)
at org.mortbay.util.Pool.newPondLife(Pool.java:368)
at org.mortbay.util.Pool.get(Pool.java:304)
at org.mortbay.util.ThreadPool.run(ThreadPool.java:368)
at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:616)
However, it doesn’'t really make sense, as the stats plugin reports normally memory usage (under 500MB for several days), and the machine is very lightly loaded. Our vmoptions file is:
-Xss128k
-Xmx2500m
Wildfire 2.5.1 on RHEL 4 running on a dual CPU Dell PowerEdge with 8GB of RAM.
I’‘m not planning on a Wildfire restart if I can help it (we’'ve had a lot of downtime lately, which is making our user base very testy), but is there anyway to restart the Jetty app server portion of Wildfire without killing anything else? Thanks.
“unable to create new native thread” is very clear for me.
I assume that you did not hit the max-threads-limit (the JVM should be able to use 256 or 512 MB as native memory to monitor the threads). How many threads are created when this error does occur?
The kernel itself anyhow should use 2 MB for each thread no matter what you specify with -Xss, so I assume that -Xss96k or -Xss64k will not help you in this case.
Compiling the kernel after editing the right[/i] files (limits.h and some other files which define the thread stack size) could[/i] solve your problem. If one posts the details here it’'s fine, but this is not a linux forum.
I’‘m sorry - I should have been clearer - yes, I’‘m very clear about what the error means, just unsure how to solve the problem, or why it happened given our memory config. I didn’‘t get a chance for a thread dump when this occured, so I don’‘t know the answer to your original question (we are monitoring this now, but the SSL admin port is hung, and we won’'t be restarting Wildfire until this weekend at the earliest).
Thanks for the info on the 2MB thread size in linux itself. At this point, we’'ve been tweaking the memory config of the VM a lot… I may have to ratchet down the memory allocated to the server in hopes of solving this another way. Recompiling the kernel is seen as a last resort for our environment. Thanks for the info.
I did not think about a javacore but a “ps -m” to display all threads. I’'m not sure whether the -m option is available with RHEL 4, /proc could also stores the number of threads, or there may be a complete other way to access it on linux.
It seems that this is linux specific and will not happen with any other operating system, but that does not really help you.
edited: If you have the option to use a 64 bit JVM it may help if the memory area for the threads was doubled, but it is not widely used.