Out-of-memory errors through sustained pre-authenticated connections

Description
The Openfire server is susceptible to out-of-memory errors if clients (by accident or on malicious purpose) keep their XMPP connections open after a failed login.

The steps for establishing an XMPP session are:

  1. Create a TCP connection to the XMPP server (usually on port 5222).
  2. Exchange supported features and negotiate authentication mechanism.
  3. Perform TLS handshake and perform authentication.

If the authentication fails, a well-behaving client is expected to close the TCP connection.
But if the client does not close the TCP connection, the XMPP server will start the server ping mechanism (even though the session was not authenticated). As long as the client responds with pongs, the TCP connection is kept open.
For example, some versions of the .NET client library MatriX do not close the TCP connection properly in such scenarios.

If the client attempts to reconnect, an additional TCP connection is opened for each reconnection attempt.

Those unauthenticated connections consume memory on the XMPP server (several hundred KB each).
When enough unauthenticated connections have been created, the XMPP server runs out of memory.

In the case we observed, the problem was caused by a faulty client, but any attacker might easily do the same. An attacker might reduce the client-side memory footprint well below that of a legitimate client, thus running a DoS attack against an XMPP server with relatively low client-side resource consumption.

Mitigation
Openfire should only use ping mechanisms (client-side or server-side) after the session was authenticated successfully.
Openfire should enforce that unauthenticated TCP connections are closed after a certain timeout (regardless of what other data, e.g., pings or other packets, the client sends).

Environment
Openfire 4.4.2 on OpenJDK 8, Oracle 12, CentOS 7.
TLS 1.2.
Authentication mechanism SCRAM-SHA-1.
Server ping enabled.

1 Like

Thanks for your report. I’ve raised this as OF-1914

I agree with the timeout for sessions that have failed to authenticate, but I’m unsure why there’s a requirement to start pings only after authentication is successful. Doing a ping during a SASL handshake would be pretty inconvenient, but I’m not seeing the immediate relation to the issue that you’re describing. Could you elaborate please?

The problem with the current ping implementation in relation to this issue is:

  • The server always responds with a pong, even if the client sends the ping before, during, or after a (failed) SASL authentication. I would expect a properly implemented client to start pinging only after successful authentication. But if a faulty client implementation starts pinging regardless of the SASL authentication state, then the server just plays along and keeps the connection open.

Of course, if the server were to close any non-authenticated connection after a timeout, the ping issue would become irrelevant.