That is a hurtful comment. I agree with you. There is nothing we can say, not even H, to make the feeling from that comment go away.
Your H is not your H. He's been possessed by this other man. The best thing you can do is stay away. He may throw another ugly comment at you if you're too close in range.
AS gave you great advice.
For what it's worth, your looks (and i'm positive you are NOT ugly) are not why he strayed. Put that all aside and work on your self esteem. Does your IC practice Behavioral Cognitive Therapy?
M 42 H 39 T10 (-2yrs separation) S8 D5 DD 7/30/11 (EA&PA) Reconciled 6/2013 Separation in works 1/2017