You are not "kicking him out of the house." You are setting a personal boundary, and letting him know that there are certain acceptable levels of courtesy and behavior for him living with you. Having an affair, staying out all night, lying to you, and treating you rudely are not acceptable to his wife.
