Today, we ran into a deadlock that involved a HTTP Binding session. I’'ve attached the deadlock report from the thread dump.
The bottom two threads are the most interesting ones. Of those two, the first thread locks on an object that the second wants to lock, and vica versa. Deadlocked. The two classes that are involved are the HttpSession class (used to implement HTTP Binding) and BaseTransport (used in the gateway plugin).
The HttpSession sends out a packet to the gateway plugin. This plugin aquires a lock during processing.
While the gateway plugin is still processing the packet, the HttpSession gets its ‘‘closeConnection()’’ method called. This method is synchronized, which causes the HttpSession to lock on itself. As part of its execution, the closeConnection() method causes another packet (presence unavailable) to be sent to the plugin. Before the plugin can start processing this packet, it has to wait for the first lock to be released.
In the mean time, the first packet causes the plugin (that’‘s still locking), to send a packet back to the user. This packet gets delivered to HttpSession’'s ‘‘deliver()’’ function, which is synchronized. As the HttpSession instance was just locked by the ‘‘closeConnection()’’ method, a deadlock occurs.
We think that HttpSession uses a locking strategy that’'s very (too) generic: The entire object is locked by a number of methods. This prevents other methods from being run simultaneously. Deadlocks are waiting to happen if the first method causes the second method to be executed (SessionListeners that get fired in one method can be expected to generate a call to another method, for example). The locking mechanism of HttpSession might be improved by replacing synchronized methods by usage of synchronize blocks that lock on a (very) specific object.
Please note that we run an adjusted version of the gateway plugin, so line numbers in the attached file might be off. We do use a similar locking mechanism as the ‘‘official’’ plugin though. But this is besides the point, as the synchronization as used in HttpSession can cause problems like these in any plugin that interacts with the client by sending/receiving packets.