This is a tough part. I think the toughest part. But, you have to pull back, give him space, and focus on YOU. Understandably, you are giving him too much of you mental real estate. As hard as it may be, focus on you and taking care of yourself. You have no control over what he does,thinks, feels. Let that go. It's hard, but necessary.
What can you do to make yourself happy and keep yourself occupied? Exercise always works for me. Cooking a nice meal for me and the kids is great too. Get busy.