No worries. I think at this point in time, your H is suffering from feelings of rejection from OW. He obviously thought it was something completely different from reality.
Just give him some space. But the best thing you can do right now is show him, he is not the end all be all of your life. He is not the center and that you are a happy person with or without him. This works on two levels I think: 1.) When you GAL, it shows him you can move on and you can have a life without him, which freaks them out. 2.) It takes the pressure off of them....lets them know they aren't responsible for making their spouse happy.....