Is there a way you can try to snip this cord so to speak to him for a couple months? Perhaps you can just immerse yourself in the pregnancy. The delivery, the newbornness, and the month or so that follows just long enough to get you through this. Deal with the rest later. Shelve it. I know that's so much easier said than done. But having a kid actually taught me how to shelve things just a little bit better. Not HUGE things like this, but ... I just wish for you some enjoyment of this time. Don't let him take it. He is confused and has no idea what he's in for. Forget an affair. Forget an other woman. Forget his confusion, his trips, his potential lies. Just grab hold of some higher version of your husband - some part of him that you knew and loved and hold on to that right now while you brace yourself for a really huge transformation. See if you can do that so you can enjoy this and be in the present. You can be in touch with that right now even if he can't.
You can follow this and really just follow it blissfully but do it without him. He may get drawn in a little. "I'm really feeling myself prepare for the baby." Say it. If he can't handle it, he won't. Nothing you say or do can stop this baby from coming now. You can't protect him from that any longer. She's coming whether he likes it or not. Go inward, and just connect with your inner self. Your bliss. Your knowing. Your place of love and fearlessness.
If anything, you will have a full experience. If anything else, your husband may be drawn in through your strength and your confidence. But have no attachment to that. Just take care of you.
I feel like virtually holding your hand right now.
You are on the right track and you're doing great.