I’d recommend upgrading to the latest Openfire release. As of now, it’s 3.8.2. I haven’t experienced this problem before myself, but you shouldn’t need to be manually editing config files or forcing system properties override via the Systems Properties box… the radial button in Log Viewer should do the trick just fine (granted everything is working well).
Just reaching… but I would think the problem isn’t the setting of the property value, since that’s trivial and done in java memory (read in from the text config file and manipulated by the Log Viweer’s radial buttons), but rather the problem is someplace else on your system. Are you getting warnings or error logs or have those stopped too? Do they show any indication of what may be going on with your Debug log? Try clearing the Error and Warnings logs and then go set the Debug option to on and check to see if anything populates the errors, etc.