The Openfire server is susceptible to out-of-memory errors if clients (by accident or on malicious purpose) keep their XMPP connections open after a failed login.
The steps for establishing an XMPP session are:
- Create a TCP connection to the XMPP server (usually on port 5222).
- Exchange supported features and negotiate authentication mechanism.
- Perform TLS handshake and perform authentication.
If the authentication fails, a well-behaving client is expected to close the TCP connection.
But if the client does not close the TCP connection, the XMPP server will start the server ping mechanism (even though the session was not authenticated). As long as the client responds with pongs, the TCP connection is kept open.
For example, some versions of the .NET client library MatriX do not close the TCP connection properly in such scenarios.
If the client attempts to reconnect, an additional TCP connection is opened for each reconnection attempt.
Those unauthenticated connections consume memory on the XMPP server (several hundred KB each).
When enough unauthenticated connections have been created, the XMPP server runs out of memory.
In the case we observed, the problem was caused by a faulty client, but any attacker might easily do the same. An attacker might reduce the client-side memory footprint well below that of a legitimate client, thus running a DoS attack against an XMPP server with relatively low client-side resource consumption.
Openfire should only use ping mechanisms (client-side or server-side) after the session was authenticated successfully.
Openfire should enforce that unauthenticated TCP connections are closed after a certain timeout (regardless of what other data, e.g., pings or other packets, the client sends).
Openfire 4.4.2 on OpenJDK 8, Oracle 12, CentOS 7.
Authentication mechanism SCRAM-SHA-1.
Server ping enabled.