I’m not sure I completely understand what you’re asking for, but I’ll take a shot at it.
If you only want users and groups from the unix OU, you can set your BaseDN to ou=unix,dc=example,dc=com.
We use Active Directory, so I’m less familiar with how OpenLDAP normally defines objects, but AFAIK, it’s impossible to define a filter to say “search dc=example,dc=com for group objects that are in ou=unix,dc=example,dc=com.” Why? Because that info is normally only in the CN, and filters are not applied to the CN. So, you will need some identifying attribute besides the CN that you can filter for.
The way AD does it, each object that is a member of a given group gets its own attribute called “memberOf” with the value set to the CN of the group that object is a member of (there can be many “memberOf” attributes for a given object). The way I solve this problem is to create a global JabberAccess group, and all users and groups that need access to Openfire are set as members of that global group. Then I set my searchFilter and groupSearchFilter to include “(memberOf=CN=JabberAccess,OU=…)”.
Hope this helps!