I would have a hard time with this if any strange person were trying to get this close to my kids -- the fact that xW's BF is also the OM/adultery partner just makes it an order of magnitude worse.
I don't think I will ever get to a point where this won't bother me.