Which IDE are you using? Remember that Jive Messenger requires Java 1.5 so be sure to use an IDE that supports such version.
If you want to build the server you can execute the ant “jar” or “all” task which will generate a new folder named “target”. Inside of that folder you will find the bin folder from where you can start the server. If you pass the -debug parameter to messenger.bat then the server will start in debug mode so you can hook a Java debugger to the server. On the other hand, if you plan on running the server inside your IDE you will need to include the jars under the lib folder in your classpath (of your project) and then lanch org.jivesoftware.messenger.starter.ServerStarter -DmessengerHome=C:\java\Messenger. Where C:\java\Messenger holds the conf, plugins and resources folders generated with ant -jar.
Let us know how it goes.