HttpBindManager creates its two BOSH ServerConnectors with a default Jetty HttpConfiguration.
Jetty’s headerCacheSize defaults to 1024, and each HttpConnection builds an ArrayTrie of that size for its field cache on the first request: a char[] of 1024 x 48 chars, about 96 KB, held for
the life of the connection. BOSH clients hold one or two long-lived connections each and repeat only a handful of short headers, so the cache saves nothing and costs about 1 GB of live heap per 10000 connected clients. Openfire 4.9.2 with Jetty 10.0.18; the same code is on main. Found while following up the HttpBindBody ThreadLocal report at
HttpBindBody keeps one XMPPPacketReader per BOSH request in the worker thread’s ThreadLocal map the measurements below were taken with that fix already applied.
Effect
On a test server (4 vCPU, 16 GB, Openfire heap capped at 2.2 GB) holding 10000 BOSH clients for one hour per column:
| header cache 1024 (default) | header cache 0 | |
|---|---|---|
| Open BOSH connections | 6300 to 8400 | 8000 to 9000 |
| BOSH requests per second | 600 | 722 |
| Openfire CPU seconds per 1000 requests | 0.30 | 0.31 |
| Live heap after a forced full GC | 796 MB | 231 MB |
| Committed heap | 2190 MB (the cap) | 588 MB |
| G1 compactions in the hour | 2 (657 ms, 518 ms) | 0 |
| Young collections / longest pause | 75 / 207 ms | 376 / 188 ms |
| Client-side median latency / p95 | 645 ms / 5.6 s | 639 ms / 5.6 s |
The heap dump taken during the first hold showed the large char[] instances owned by
org.eclipse.jetty.http.HttpParser$FieldCache._cache → org.eclipse.jetty.util.ArrayTrie._table,
one per HttpConnection.
Reproduction
Connect a few thousand BOSH clients and keep them polling, then
jcmd <openfire pid> GC.run followed by jcmd <openfire pid> GC.class_histogram | head -20: the[C line grows by about 96 KB per open connection and does not shrink while the connections stay open. A heap dump confirms the owner chain above.
Fix
Expose the size as a property with the current default, so nothing changes for existing
deployments, and large deployments can lower it:
// HttpBindManager, on both the plain and the TLS BOSH connectors
httpConfig.setHeaderCacheSize( JiveGlobals.getIntProperty( "httpbind.header-cache.size", 1024 ) );
Setting it to 0 disables the cache; a miss means the header is parsed the normal way, which is also
the path for every first-seen header. The documented purpose of the cache is to avoid re-creating
HttpField objects for headers repeated on a keep-alive connection; BOSH traffic repeats few, and the CPU per request above did not change.
Verification of the fix
The right-hand column above is the same server, same load, with setHeaderCacheSize( 0 ) on both connectors: heap off the cap, no compactions, same CPU per request and same client latency. The cross-session isolation test (100 concurrent sessions, marked stanzas, one malformed body in seven): 122626 requests over two 5-minute runs with both changes in place, every marked stanza returned to its own session, none to another session or JID, 8670 malformed bodies all answered HTTP 400 with no session terminated.