Ask him for specifics. You can't possibly know what he is talking about without details. Don't make yourself crazy trying to figure it out. As you know all too well he will either tell you or not. If you ask and he explains then you have something to go on. If he doesn't you haven't really lost anything, have you?
Be careful about the speculating. There is only one person who really knows what he is thinking or doing or feeling and that is your H and he obviously isn't willing to let you in on it at the moment. We can play what if until the cows come home but that really gets us nowhere.
Take your space when you need it. It helps to clear your head and center you. Continue to live your life while doing no harm. You can do this. You have before. Keep your children close and love them up. {I know you already are. :-)}
Hang in there. Only you can determine when you have reached your enough is enough.
*hugs* ~ swl
Look not mournfully into the past. It comes not back again. Wisely improve the present. It is thine. Go forth to meet the shadowy future, without fear. {Henry Wadsworth Longfellow}