I think what you need to do now is to let him know he isn't calling all the shots. If he knows you are sitting there waiting for him then he will do anything he wants. Tell him if he wants to "be alone" then that means no phone calls or anything. I have learned from experience that when I keep myself away from my H, he tends to come around more. It is so hard to do this but it really works. Give it a try, what have you got to lose?
