Log spam: "ERROR org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic ..."

Hi,

My /opt/openfire/logs/openfire.log is constantly being spammed with the following messages, a batch of these is logged exactly once every 60 seconds:

2022.07.15 12:40:57 ERROR [pool-monitoring6]: org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic server_bytes_out
java.io.IOException: The underlying backend has no canonical path
	at org.jrobin.core.RrdDb.getCanonicalPath(RrdDb.java:1029) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.Header.validateHeader(Header.java:210) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:231) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:201) ~[jrobin-1.6.0.jar!/:?]
	at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:359) [monitoring-2.3.0.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
2022.07.15 12:40:57 ERROR [pool-monitoring6]: org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic muc_occupants
java.io.IOException: The underlying backend has no canonical path
	at org.jrobin.core.RrdDb.getCanonicalPath(RrdDb.java:1029) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.Header.validateHeader(Header.java:210) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:231) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:201) ~[jrobin-1.6.0.jar!/:?]
	at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:359) [monitoring-2.3.0.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
2022.07.15 12:40:57 ERROR [pool-monitoring6]: org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic server_bytes_in
java.io.IOException: The underlying backend has no canonical path
	at org.jrobin.core.RrdDb.getCanonicalPath(RrdDb.java:1029) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.Header.validateHeader(Header.java:210) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:231) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:201) ~[jrobin-1.6.0.jar!/:?]
	at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:359) [monitoring-2.3.0.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
2022.07.15 12:40:57 ERROR [pool-monitoring6]: org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic proxyTransferRate
java.io.IOException: The underlying backend has no canonical path
	at org.jrobin.core.RrdDb.getCanonicalPath(RrdDb.java:1029) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.Header.validateHeader(Header.java:210) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:231) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:201) ~[jrobin-1.6.0.jar!/:?]
	at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:359) [monitoring-2.3.0.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
2022.07.15 12:40:57 ERROR [pool-monitoring6]: org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic conversations
java.io.IOException: The underlying backend has no canonical path
	at org.jrobin.core.RrdDb.getCanonicalPath(RrdDb.java:1029) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.Header.validateHeader(Header.java:210) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:231) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:201) ~[jrobin-1.6.0.jar!/:?]
	at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:359) [monitoring-2.3.0.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
2022.07.15 12:40:57 ERROR [pool-monitoring6]: org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic packet_count
java.io.IOException: The underlying backend has no canonical path
	at org.jrobin.core.RrdDb.getCanonicalPath(RrdDb.java:1029) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.Header.validateHeader(Header.java:210) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:231) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:201) ~[jrobin-1.6.0.jar!/:?]
	at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:359) [monitoring-2.3.0.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
2022.07.15 12:40:57 ERROR [pool-monitoring6]: org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic muc_rooms
java.io.IOException: The underlying backend has no canonical path
	at org.jrobin.core.RrdDb.getCanonicalPath(RrdDb.java:1029) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.Header.validateHeader(Header.java:210) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:231) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:201) ~[jrobin-1.6.0.jar!/:?]
	at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:359) [monitoring-2.3.0.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
2022.07.15 12:40:57 ERROR [pool-monitoring6]: org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic server_sessions
java.io.IOException: The underlying backend has no canonical path
	at org.jrobin.core.RrdDb.getCanonicalPath(RrdDb.java:1029) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.Header.validateHeader(Header.java:210) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:231) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:201) ~[jrobin-1.6.0.jar!/:?]
	at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:359) [monitoring-2.3.0.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
2022.07.15 12:40:57 ERROR [pool-monitoring6]: org.jivesoftware.openfire.reporting.stats.StatsEngine - Error sampling for statistic muc_users
java.io.IOException: The underlying backend has no canonical path
	at org.jrobin.core.RrdDb.getCanonicalPath(RrdDb.java:1029) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.Header.validateHeader(Header.java:210) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:231) ~[jrobin-1.6.0.jar!/:?]
	at org.jrobin.core.RrdDb.<init>(RrdDb.java:201) ~[jrobin-1.6.0.jar!/:?]
	at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:359) [monitoring-2.3.0.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]

Do I need to configure something? If yes, what and where? If no, is this kind of log spam normal and to be expected? I’m running Openfire 4.7.2 with the Monitoring Service 2.3.0 plugin (which is where I understand the log spam is coming from), among a few others, on a 32bit Slackware 15.0 system with JRE 1.8.0_321-b07.

If you need any more information, please let me know. Thanks! :slight_smile:

This is definitely an issue. The Monitoring Service plugin keeps data structures (RRDs) in the Openfire database (it’s a bit of a database-in-a-database construct). Your RRDs appear to be malformed for some reason (annoyingly, the error that is being reported is caused by a problem reporting that problem, which means that a bit of the context is lost).

This will probably need manual interaction to get fixed.

  1. Shut down Openfire
  2. Create a backup of your entire database (and of Openfire, for good measure)
  3. In the database, look for the table named ofRRDs. See if there is something obviously weird about that table (I’m not sure what to look for, but you might know if you see it).
  4. If all else fails, TRUNCATE (but do not delete) that table.
  5. Restart Openfire

Okay, here’s what my table looks like, from my 4.7.3 installation which had the same problem (I always keep the previous version aronud in case something goes wrong when upgrading). I have no idea how it is supposed to look, so I’m not sure if it’s actually weird or not. :person_shrugging:


Soooo … weird or not?

If I have to truncate that table, how would I do that? Is it as easy as entering TRUNCATE TABLE "PUBLIC"."OFRRDS" in the SQL statement field and then hitting “Execute SQL” (after shutting down OF and making a backup, of course)?

Hmm, looks sane enough - but maybe there is something in those byte arrays that is corrupt.

Truncating a table in HSQLDB seems to be that simple, yes:

TRUNCATE TABLE <target table>

Full documentation here: Chapter 5. Data Access and Change

Well, okay, then I’ll give it a go later on today and let you know how it went. Wish me luck! :slight_smile:

Yes, that fixed it - thank you so much, that really made my day! :+1: :sunglasses: :partying_face:
Statistics work:


Reports work as well:

Yay! :grin:

1 Like