powered by Jive Software

Gaps in message audit logs

We have a large gaps in openfire audit logs. These gaps occur at log file rotation points.

For example, the last entry in jive.audit-20070825-001.log is time stamped Aug 24, 2007 11:43:03:697 PM

And the first entry in jive.audit-20070825-002.log is time stamped: Aug 25, 2007 06:55:38:091 PM

So, most logs for 08/25 are missing.

It appears that we have this type of problem pretty frequently (once in day or two days). (Yes, the server is running and functioning during that time. It’s 3.3.2 on linux).

Has anyone seen this or can shed some light on this? Is setting “Maximum size of all files” in message audit policy can have such effect?

Thanks in advance.

Hi,

did you set the max. size to more than 2 GB?

JM-909 is the known problem about larger sizes.

LG

Nope. Max file size was set to 10M. Max size of all files was set to 1000M.

Another thing to note is that auditing is turned on for all types of packets and we do have a fair amount of data. 10M audit file would hold about 2-4 hours of data.

We had the same problem. The issue is down to the way log-rollover works. It is consistently overwriting old log files hence there are gaps in the data.

The code lists the log files to find the most recent. It assumes that the most recent log file is at the end of the list. This isn’t guarenteed to be true. It seems to work fine on Windows but consistently fails on Linux.

In more detail: The file: org.jivesoftware.openfire.audit.spi.AuditorImpl.java, line 276 does this:

File[] files = baseFolder.listFiles(filter);

Then line 283 does this to determine the most recent file and hence work out the next log file number to use.

File lastFile = files[http://files.length - 1|http://files.length - 1];

However, the javadoc for File.listFiles() states

“There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.”

So, on Linux the code retrieves a log file that is not the latest (say 001.log), increments the log number (to say 002) and thus overwrites 002.log with newer data. This gives you the gaps you see.

The simplest fix is to add the following code before line 283:

// File implements Comparable as a lexographical ordering of abstract
// path names.  Arrays.sort() uses the Comparable interface.  So this shoul
// give us an alpha sorted array of files.  This is ok, because the fil
// name format is such that an alpha sort will produce a date sort + log+
// number sort. We want a date + log number sort so that the next line of++
// code after this one retrieves the most last log file in the sequence
// Without this fix, on Linux, the list of log files is not guarentee
// to be in the correct order Arrays.sort(files);

A more robust solution might be to write code that explicitly finds the largest log file number rather than relying on an alpha sort. Or, remove the rollover code entirely and just use log4j instead - although AFAIK this doesn’t support Openfire’s exact log rollover policy.

Looking at the rest of the code base, there doesn’t seem to be any other similar mis-use of File.listFiles() although the class-loaders (JiveClassLoader and PluginClassLoader) use it which might cause different versions of the same class to be loaded on different platforms.

Code to demonstrate the problem:

import java.io.File;
import java.io.FilenameFilter;
import java.util.Arrays; public class FileListTest
{
   // arg0 is log directory, arg1 is a date e.g. 20071206
   public static void main( String[] args )
   {
      File baseFolder = new File( args[0] );
      final String filePrefix = "jive.audit-" + args[1];       // orig code
      FilenameFilter filter = new FilenameFilter( )
      {
         public boolean accept( File dir, String name )
         {
            return name.startsWith( filePrefix ) && name.endsWith( ".log" );
         }
      };
      File[] files = baseFolder.listFiles( filter );
      // orig code end       // Take a copy of the file list for later comparison
      File[] unsortedList = new File[http://files.length|http://files.length];
      System.arraycopy( files, 0, unsortedList, 0, files.length );       Arrays.sort( files ); // Fix code       // Print unsorted vs sorted. Note differences with an arrow.
      for ( int i = 0; i < unsortedList.length; i++ )
      {
         System.out.print( unsortedList[i] + "  " + files[i] );
         if ( !unsortedList[i].equals( files[i] ) )
         {
            System.out.print( " <-----" );
         }
         System.out.print( "\n" );
      }
   }
}

Example usage:

java FileListTest /opt/openfire/logs 20071205

/opt/openfire/logs/jive.audit-20071205-000.log /opt/openfire/logs/jive.audit-20071205-000.log

/opt/openfire/logs/jive.audit-20071205-001.log /opt/openfire/logs/jive.audit-20071205-001.log

/opt/openfire/logs/jive.audit-20071205-003.log /opt/openfire/logs/jive.audit-20071205-002.log <-----

/opt/openfire/logs/jive.audit-20071205-002.log /opt/openfire/logs/jive.audit-20071205-003.log <-----

LHS show the default list order returned from listFiles(), RHS shows it sorted. Lines marked with an arrow show differences which would result in log file gaps

I did create JM-1212 to fix this issue.

Thanks, getting this fix rolled into openfire would be much appreciated.

Ta,

Chris

Hi Chris,

Are you stil around on this forum to speak on this issue with the current release?

thanks!

daryl

I can say is still an issue with 3.6.1. By inspection, none of the code commits since fix it.