I think your H is right.

Get rid of the boyfriend if you want to work on the marriage with your H. Sara is right.