You did a very nice job w/this conversation. He's upset that your family is coming to town because he doesn't know if they know about the situation. They like to keep things status quo and have everyone think that things are okay. He's concerned that he will be judged for what he's done by your family. That's the stressors he's got going on w/them being there and also, he would have to keep up appearances too.
Notice how he used the word divorce to try to get you to chance your mind? They like to toss that threat around when they feel that they are being pushed up against the wall. It's their way of getting us to back off and when he saw you weren't, he slowly changed his tune. This was an excellent move on your part to stand your ground on your family visiting.
No matter what you do or don't do, right now, he'll pick it to death and tell you that you are insensitive. Bottom line, that's projection on his part. His empathy chip is broken and of course he doesn't understand why you would care about the affair. To him, it was a booty call w/no strings attached.
How can you understand what he feels or is thinking when he doesn't understand himself? If he doesn't share things w/you, you certainly can't pull out the good old crystal ball and find out. You aren't a mind reader.
Listen to what he has to say, sift through the garbage coming out of his mouth and only take away what is meaningful. Trust me, if you sit quietly and listen, you'll find out more about what your h is doing and what he is actually thinking. They do tend to spill the beans more if you show them attention and truly listen.
I do hope that you can enjoy your vacation.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.
