What have you said to him at this stage of the game?
I am not encouraging you to go either way...preserve the M or not..but I am hoping that you have expressed to him that he is making a choice between these options: Be a sexual mate to my partner; and Give up because I don't want to try.
I think it is dreadful that he is choosing what he has. I'm sorry for you and your kids. I wish I was there to take you out tonight.