it will be really hard, and even harder if she insists on having OM around - do you know if i have (even though not Scotland) have any legal grounds to deny him access.
One way to keep this from happening, even when you are not in the house (because even if you throw him out and ask him never to come back it seems your wife can "invite" anyone on the premises she desires...especially if you aren't there) would possibly be to file a personal protection or restraining order of some kind against the guy. That his presence on your property is and would create a hostile situation and a threat to you. That [om] is doing it intentionally with the intent to harass you and egg you on into a violent response. Maybe pleading that it's taunting. You may not be able to get something like this for walking around town absent a specific threat but on your own property (even though it's joint with your wife) might be obtainable. The "treat" is his presence on your property even though he's been specifically asked NOT to enter it. (you may need to make a specific notice to advise OM that his entering your property is perceived by you as a threat and could result in you defending yourself [your "defending yourself" may sound like you are going to beat him up but it really means - pursuing legal recourse to defend yourself from his presence.)
Property rights are taken seriously by the courts and the US right to peaceful/quiet enjoyment of your property derives it's roots in British Common Law.
In the US you could petition for exclusive use of the marital residence and seek to have a judge remove the other spouse for cause. Bringing an affair partner into the home against the wishes of the other spouse could be a persuasive factor in such motions.
The internet is 90% complaining and entitlement and I hate it because I deserve better!