mdu, I'm sorry for this latest blow. I'm glad you two are talking about it openly and honestly at least.
Personally, I'm less concerned about the contact itself than I am with the combination of your husband's either lack of concern for your feelings on the matter, and/or his utter lack of awareness that makes him think his currently-still-fragile marriage can survive such an arrangement.
He's either uncaring or a fool, THAT'S what concerns me.
Pyromaniacs shouldn't work around matches and kerosene.