I have to admit I'm agreeing with the others that are asking you why you are continuing to see this OM (even if he is just a friend). People will often say things like what your H has...that it's not a problem, or he doesn't mind, or whatever....when they don't want to admit that it IS an issue for them. Many people don't want to appear jealous or insecure, his telling you that it bothers him runs the risk for him of appearing either or both.