He's obviously starting to feel the sting of losing his family and the respect of his children.
It's a cheap shot for him to accuse you of being happy about his shitty relationship with the kids - of course he's still not ready to face his responsibility for that.
Don't let him sucker you into any tit for tat on that stuff. You did a good job in this communication. And if he brings up you avoiding him again, just tell him - "Is it really any surprise to you that I prefer not to see my exd-husband who is having a baby with another woman????"
