Is there a way to limit which users can broadcast messages?

I see the option to block it all with the connection manager plugin but not to limit it on a per user basis?

If you are talking about Broadcast plugin, there is such property to limit broadcasting only to allowed users. But some clients have their internal broadcasting capabilities (which in fact is a multiple simple message sending) which cant be controlled by a server.