LOL guess you answered my question in an edit. Do you have the JDK or JRE installed? JDK includes some useful tools for dumping memory. (look for jmap)
I’‘ve got JDK 1.5.0 currently, I’'ll wait for the memory to load up tomorrow and do a dump…
Thanks again
Josh
Message was edited by: JoshM
If you have JDK 1.5.0 you should have jmap =) All you need to do is find the process id of your openfire server and run jmap -histo
(thanks to srt and juancarlos for explaining this) do this when your server has a lot of memory usage so we can see what might be taking all of it up.
Hi jadestorm,
it seems you are using JDK 5 which requires additional parameters to be passed to the java process to allow jmap to attach to it. Upgrade to Java 6 and it will work without any additional settings.
=Stefan
I am using java 6 =/
From the admin console: JVM Version and Vendor: 1.6.0 Sun Microsystems Inc. – Java HotSpot™ Server VM
What occurs is this:
/usr/java/jdk1.6.0/bin/jmap -histo 6492
6492: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
One interesting note. /usr/java/jdk1.6.0 is my “real” java 6 JDK install. /opt/openfire/jre is a bundled java 6 which is what openfire is actually running under. (it does not include the full JDK, just JRE) Is there any reason why it -must- be the same tree? Or does the JRE not have the proper things built into it?
Can you post some more information and the error message then?
I get a similar error when i am not using the same user openfire runs as when requesting the history (like using root instead of openfire):
root@eins:~# jmap -histo 21736
21736: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
root@eins:/etc/subversion# su - openfire
openfire@eins:~$ jmap -histo 21736 num #instances #bytes class name
--------------------------------------
1: 47721 7319672 <constMethodKlass>
2: 56877 6189472 [C
3: 47721 5735912 <methodKlass>
Edit: sorry didn’'t see your last post - which user are you running openfire as?
Message was edited by: srt
Awesome, that was is! Was running jmap as root and running openfire as daemon. =) Thanks! (learn somethin’’ new every day!)
I would find it better if igniterealtime would do bugfix releases if they find something like memory leaks or security issues.
Who’‘s igniterealtime? Do you mean jive? =) If you are referring to the leaks found in openfire itself, yeah… I agree. If you are referring to the IM Gateway plugin, this is the only thing keeping 1.1.0 from being released. Once I can figure out what’'s going on… =/
Yes sorry I meant jive… With gateway 1.1.0 it’'s not a problem because therefore it is a beta and not a gold release.
I ran JMAP on our Openfire this morning while it was idling at about 300MB allocated. I am still running JDK 1.5.0 so I’‘m not sure if it output anything usable, but could someone take a look at the top sized Classes and let me know if there’'s anything interesting in here?
Thanks,
Josh
Size Count Class description
169642144 5301317 java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
27354880 264871 char[]
13306040 31140 byte[]
10483944 124379 java.util.HashMap$Entry[]
8928216 372009 java.lang.String
7214456 50693 * ConstMethodKlass
4393176 82297 java.lang.Object[]
4269720 106743 java.util.HashMap
3853960 83874 * SymbolKlass
3653504 50693 * MethodKlass
3002312 4757 * ConstantPoolKlass
2674112 167132 java.util.concurrent.ConcurrentLinkedQueue$Node
2201712 91738 java.util.HashMap$Entry
2042656 63833 java.util.LinkedHashMap$Entry
2025648 42201 org.jivesoftware.openfire.roster.RosterItem
1917856 119866 java.util.concurrent.LinkedBlockingQueue$Node
1872888 78037 java.util.ArrayList
1845904 4757 * InstanceKlassKlass
1700208 4034 * ConstantPoolCacheKlass
1613888 50434 java.util.concurrent.ConcurrentHashMap$Segment
1520336 13054 int[]
1389760 86860 java.util.HashSet
1243128 51797 java.util.concurrent.ConcurrentHashMap$HashEntry
1214712 50613 java.util.concurrent.locks.ReentrantLock$NonfairSync
1158984 16097 net.sf.jml.impl.MsnContactImpl
1131800 50941 java.util.concurrent.ConcurrentHashMap$HashEntry[]
994688 31084 org.xmpp.packet.JID
988872 41203 java.util.LinkedList$Entry
848280 35345 java.util.LinkedList
845664 17618 java.util.LinkedHashMap
835880 3102 * MethodDataKlass
559632 11659 org.jivesoftware.openfire.gateway.protocols.msn.MSNBuddy
553288 6609 short[]
546752 34172 java.util.HashMap$KeySet
496360 12409 java.util.WeakHashMap$Entry
486816 5071 java.lang.Class
375264 23454 net.sf.jml.MsnClientId
365472 15228 org.dom4j.tree.DefaultElement
344784 819 sun.misc.CacheEntry[]
334880 10465 net.sf.jml.protocol.incoming.IncomingNLN
331696 7357 java.lang.Object[]
322656 10083 net.sf.jml.protocol.outgoing.OutgoingPNG
307904 9622 net.sf.jml.protocol.incoming.IncomingQNG
273488 17093 java.util.LinkedHashSet
261600 16350 net.sf.jml.Email
259968 16248 net.sf.jml.impl.MsnUserPropertiesImpl
252192 3154 java.util.concurrent.ConcurrentHashMap$Segment[]
240608 7519 net.sf.jml.protocol.incoming.IncomingUBX
169328 10583 org.dom4j.tree.DefaultText
168288 7012 java.util.Hashtable$Entry
166320 2310 com.mysql.jdbc.ConnectionProperties$BooleanConnectionProperty
161952 5061 org.jivesoftware.util.LinkedListNode
140040 2942 java.lang.String[]
136160 3404 java.math.BigInteger
I forgot to restart the server last night and it died on us… kill -9 this morning to restart it.
Error Log:
2007.07.12 05:48:10 org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHandle r.java:109)
java.lang.OutOfMemoryError: Java heap space
2007.07.12 05:48:12 org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHandle r.java:109)
java.lang.OutOfMemoryError: Java heap space
2007.07.12 05:48:12 com.jivesoftware.openfire.enterprise.stats.StatsEngine$SampleTask.run(StatsEngin e.java:328) Error sampling for statistic sessions
org.jrobin.core.RrdException: Bad sample timestamp 1184244480. Last update time was 1184244480, at least one second step is required
at org.jrobin.core.RrdDb.store(Unknown Source)
at org.jrobin.core.Sample.update(Unknown Source)
at com.jivesoftware.openfire.enterprise.stats.StatsEngine$SampleTask.run(StatsEngi ne.java:322)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2007.07.12 05:48:12 com.jivesoftware.openfire.enterprise.stats.StatsEngine$SampleTask.run(StatsEngin e.java:328) Error sampling for statistic sever_sessions
org.jrobin.core.RrdException: Bad sample timestamp 1184244480. Last update time was 1184244480, at least one second step is required
at org.jrobin.core.RrdDb.store(Unknown Source)
at org.jrobin.core.Sample.update(Unknown Source)
at com.jivesoftware.openfire.enterprise.stats.StatsEngine$SampleTask.run(StatsEngi ne.java:322)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2007.07.12 08:42:02 org.jivesoftware.database.SchemaManager.checkSchema(SchemaManager.java:251)
java.sql.SQLException: Duplicate column name ‘‘room’’
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2975)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1600)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1695)
at com.mysql.jdbc.Connection.execSQL(Connection.java:2998)
at com.mysql.jdbc.Connection.execSQL(Connection.java:2927)
at com.mysql.jdbc.Statement.execute(Statement.java:535)
at org.jivesoftware.database.SchemaManager.executeSQLScript(SchemaManager.java:348 )
at org.jivesoftware.database.SchemaManager.checkSchema(SchemaManager.java:248)
at org.jivesoftware.database.SchemaManager.checkPluginSchema(SchemaManager.java:11 1)
at org.jivesoftware.openfire.container.PluginManager.loadPlugin(PluginManager.java :378)
at org.jivesoftware.openfire.container.PluginManager.access$200(PluginManager.java :46)
at org.jivesoftware.openfire.container.PluginManager$PluginMonitor.run(PluginManag er.java:916)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101 (ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodi c(ScheduledThreadPoolExecutor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Schedu ledThreadPoolExecutor.java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
It’‘s so hard to tell what’‘s going on . . . I see a lot of JML in there though. But it’‘s also hard to tell if that’'s just lots of activity or something. That second log unfortunately is only indicating things failing -because- of memory issues, not what might be causing them. Did I already ask what OS you are running this on?
Gentoo Linux 2.6.17 r7 - i686 Dual Core AMD Opteron™ Processor 280.
It’'s running in a virtual machine (vmware server 1.0.0).
I have no java skills whatsoever so i am pretty lost on this one…
Does anyone know if there’'s a such thing as… like… dumping a state file that JProfiler could later examine? I see support for .hprof and .jps, however you create those… I think that might be the ultimate because then I could open it up in my debugger and see what maps to what. (at least I hope so)
with JDK 6 use
jmap -dump:format=b,file=heap.hprof <pid>
(again as the same user openfire is running as)
Awesome, thank you!
It looks like there’‘s a chance you can accomplish it with JDK 1.5.0 with -heap:format=b (i have no idea if it supports the file attribute, I can’'t get it to work at all on my mac)
Hi Daniel,
while jmap works with JRE 1.5 it causes invalid heapdump files so it’'s no fun to analyze them. For me the JRE did continue running and to analyze a file of a 768 MB heap did need 3 GB memory on the client so I had some fun to find a client with enough memory.
LG
Yikes =)
Well I did get my hands on a dump but I’‘m confused about what I’'m seeing…
The class with the top number of instances is:
java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
and in this case there are 3,763,347 instance of it taking up around 100MB of memory. It’'s by far the highest. Next down is char[] with 36,754 instances and 4M of memory.
The most bizarre thing to me is that when I try to trace back the GC root of any one of the AbstractQueuedSychronizer instances, the thing keeping it alive is … an AbstractQueuedSynchronizer instance. I can keep tracing this over and over again and keep getitng more of them as I go back. If I ask jprofiler to find me the root it crashes. =/
I don’'t get it. Not sure what to look for. =/
I doubt this is the issue, but what database is everyone using that is seeing the leaks?
For that matter, is there anyone but me that is -not- seeing the leaks?