Can I specify which fleets can see my site?

Yes, when creating or editing a site, you specify under Visibility who sees it. With Visible to all fleets on Depotcharge, every fleet finds it, whereas Only visible to fleets with an agreement shows it only to your partners.

Note that visible does not mean usable, because only fleets with an agreement are permitted to charge.