powered by Jive Software

HTTP ERROR 500 "security-audit-viewer.jsp"

#1

I’ve installed an Openfire server, everything works fine except for when i try to go to the “Security Audit Viewer”.

i get the following error:

HTTP ERROR 500

Problem accessing /security-audit-viewer.jsp. Reason:
Server Error

Caused by:

java.lang.OutOfMemoryError: Requested array size exceeds VM limit

env setup:
Openfire 4.4.4
jdk1.8.0_231 (oracle) (also tried the builtin jre, i get the same result)
centos 7
OPENFIRE_OPTS -Xmx: 1024m (i also tried with 3G) -Xms: 512m (i also tried running the server without this, i get the same result).

Thanks!!!

#2

Are you getting this on a new install or is this an upgrade with lots of history stored in the database?

#3

That doesn’t sound right. :worried:

Can you create a heap dump and analyze what’s eating up all the memory? There are helpful Java options that you can set that will cause a heapdump to be created automatically when an OutOfMemoryError occurs.

#4

this is a “new” install, the server is clean and fresh but i used a db dump files from an “old” openfire version (4.2.3), the dump files only holds the default openfire settings.
this happens on both Oracle XE (11.2.0) + ojdbc7.jar and ojdbc8.jar (i removed the other file while testing this) and MSSQL (using new connection string).

#5

hope this helps, also please notice my notes from the other reply :slight_smile: thanks

2019.12.02 08:25:09 WARN [Jetty-QTP-AdminConsole-60]: org.eclipse.jetty.server.HttpChannel - /security-audit-viewer.jsp
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at oracle.jdbc.driver.Accessor.setCapacity(Accessor.java:828) ~[ojdbc7.jar:12.1.0.1.0]
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:804) ~[ojdbc7.jar:12.1.0.1.0]
at oracle.jdbc.driver.OracleStatement.check_row_prefetch_changed(OracleStatement.java:2965) ~[ojdbc7.jar:12.1.0.1.0]
at oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:3581) ~[ojdbc7.jar:12.1.0.1.0]
at oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchMoreRows(InsensitiveScrollableResultSet.java:1008) ~[ojdbc7.jar:12.1.0.1.0]
at oracle.jdbc.driver.InsensitiveScrollableResultSet.absoluteInternal(InsensitiveScrollableResultSet.java:972) ~[ojdbc7.jar:12.1.0.1.0]
at oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:572) ~[ojdbc7.jar:12.1.0.1.0]
at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:1160) ~[commons-dbcp2-2.6.0.jar:2.6.0]
at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:1160) ~[commons-dbcp2-2.6.0.jar:2.6.0]
at org.jivesoftware.openfire.security.DefaultSecurityAuditProvider.getEvents(DefaultSecurityAuditProvider.java:149) ~[xmppserver-4.4.4.jar:4.4.4]
at org.jivesoftware.admin.servlet.SecurityAuditViewerServlet.doGet(SecurityAuditViewerServlet.java:39) ~[xmppserver-4.4.4.jar:4.4.4]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) ~[jetty-servlet-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) ~[jetty-servlet-9.4.18.v20190429.jar:9.4.18.v20190429]
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129) ~[sitemesh-2.4.2.jar:?]
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77) ~[sitemesh-2.4.2.jar:?]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[jetty-servlet-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.jivesoftware.util.LocaleFilter.doFilter(LocaleFilter.java:73) ~[xmppserver-4.4.4.jar:4.4.4]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[jetty-servlet-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.jivesoftware.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:49) ~[xmppserver-4.4.4.jar:4.4.4]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[jetty-servlet-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:226) ~[xmppserver-4.4.4.jar:4.4.4]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[jetty-servlet-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:234) ~[xmppserver-4.4.4.jar:4.4.4]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) ~[jetty-servlet-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[jetty-server-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[jetty-server-9.4.18.v20190429.jar:9.4.18.v20190429]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) ~[jetty-server-9.4.18.v20190429.jar:9.4.18.v20190429]

#6

What you provided is a stack trace, not a heap dump. The stack trace tells us what operation was failing at the time the out-of-memory was hit (it was creating a database result), but it does not tell us what is being kept in memory.

How many rows has the ofSecurityAuditLog table in your database?

#8

ofSecurityAuditLog has 16 rows.
heap dump attached, hope this helps.
Thanks
heapdump.bin.tar.gz (23.4 MB)

#9

i found the issue, i stared manually clearing records from the 'ofSecurityAuditLog` table until it worked, i also generated new lines in that table (by changing the servers settings), as long as that table has 9 or less records im able to view the " Security Audit Log Viewer" once i generate the 10th record thats when i get the original 500 error. JVM memory is on 20% usage while this happens.

any idea/thoughts of a solution?

#10

The heap dump that you provided does not show any kind of memory issue. Was it created while you were having the Out-of-Memory problem?

We’ve recently added pagination to that page. Maybe that introduced the issue. It’s working fine for me though.

@gdt Do you happen to have any thoughts on this?

Maaaybe this isn’t a “regular” out-of-memory, but some kind of problem with an array being initialized to MAX_VALUE (which could be bigger than allowed), or something like that? Something like that could cause a driver-specific issue, I guess.

#11

The heap dump that you provided does not show any kind of memory issue. Was it created while you were having the Out-of-Memory problem?

yes it was

#12

This is an odd one - and almost certainly an odd interaction with the Oracle JDBC driver. Right now the best I can come up with is
a) Try a new JDBC driver (12.1.0.2) though I have little faith that will make any difference
b) Fix the DefaultSecurityAuditProvider so that it takes null for the number of events (as documented) without crashing, and pass in null instead of Integer.MAX_VALUE from SecurityAuditViewerServlet

Greg

1 Like
#13

Ps. An entirely unrelated note, Oracle drivers are now on Maven Central (https://medium.com/oracledevs/oracle-jdbc-drivers-on-maven-central-64fcf724d8b) so we could include them in our distribution (my understading of the licence @ https://search.maven.org/artifact/com.oracle.ojdbc/ojdbc8/19.3.0.0/jar suggests it’s OK to do so).

1 Like
#14

I have created this issue in our bugtracker for this problem: https://issues.igniterealtime.org/browse/OF-1936

#15

I’ve put in a MR for this - I don’t /know/ it’s fixed it (as I couldn’t reproduce it without installing Oracle) but strongly suspect it has.

Greg

#16

i just wanted to confirm a few things…
i can see the same results with a MSSQL DB but the “workaround” (clearing the table), doesnt work on that DB. also i already tried updating the JDBC driver… that didnt help as well

#17

Thanks for the fix, @gdt! I’ve merged and cherry-picked it to the 4.4 branch.

@prili, Gregs fixes are now available in thelatest nightly build. Give that a try and see if the problem disappears.

#18

yes, looks like the issue has been resolved.
i can confirm that it works from both oracle DB and MSSQL.

do you have any idea when this fixed will be merged and release into a stable version?

and thanks for the all help and finding a quick solution!!! gr8 job!!!

#19

If I were to guess, I’d say we’d do another release in a couple of weeks.