In theory it should work, but as installer has to do a few dozens of database updates since 3.8.0 and many files and libraries have changed, it can fail.
Sounds a bit like https://issues.igniterealtime.org/browse/OF-1647 though it should be fixed. Check if you have plugins/admin/webapp/WEB-INF/lib folder and remove it. Stop Openfire before doing that.
If it doesn’t help, you can revert back to backup and try doing incremental updates. Explanation: upgrading Openfire