You're doin' fine, girl. You put up with waaaaayyy more from your H than most would.
Yes, it's a worry, risking the other man's feelings when you may not be done with your H. But I also totally understand how nice it is for you to have the validation, after putting up with your H's BS for so long.
