The more I think about it the more it looks like a Gateway issue. At least a few of the contacts are chat requests I denied. They have never been added to my roster by myself.
We have had the same issue here, at least with my account, one of my OLD Yahoo Contacts never goes away even after attempts to remove several different times.