Filtering buddies visibility

I am trying to limit the visibility of buddies a user can see. Basically, it should do the same as the Search plugin does - filter the results on the groups the current user is a member of. My problem is that if the user does directly add a buddy then it just works. I am trying with the Pidgin 2.13 and Swift 4.0.2. I have my OpenFire Search configured to limit the search result on the groups and it works as long as the user is doing a search.
I have a feeling that when the user simply does Add Buddy in Swift or in Pidgin then the request goes not through the Search plugin, but takes a different path. Can anyone explain to me what is that different path. I can eventually try to put the same filter that is in the Search plugin on that path, too.
TIA for any input.