are you running a 32 bit JRE? You may encounter a 2 GB process limit if you set Xmx to a value short below 2 GB as Xmx is only the Java heap. The JVM also has a native heap (40-500 MB) and the Permsize heap (16-128 MB), so setting Xmx to more than 1,5G will likely fail with a 32 bit JRE.
you could use a 64 bit operating system and JVM but you may want to note that a 64 bit JVM uses about - depending on the application - 1.4 to 1.9 times more memory than a 32 bit JVM. So you may have less resources available if you set -Xmx2600m within a 64 bit JRE than Xmx1500m within a 32 bit JRE.
If you really need so much memory within Openfire to handle the rosters, connections, … you could either try connection managers or install a cluster on one server.