PS - this is so hard. I agree with other posters, that the apparent uncaring is part of the mask they wear to hide the deep pain. But, until and unless he faces down those demons, instead of projecting it all onto you your marriage will be troubled.
He has to work through this stuff, and seeing him go again will be so hard. Hang on to the feelings of love and compassion you have for him. I always try and think lovingly of my h,even when he is being hurtful [not always successful at this = MLCers can be most provoking, like teenagers].
It will hurt terriblyat first, but you will feel better quicker than previous times, I think. Doesn't mean you won't still have the bad days.
You made a tough call, and have the guts to live with the outcome. If he takes the opprotunity to grow and sort himself, then you could have a great realtionship, but if not, it would always be flawed and damaged, and frankly PS you deserve better than that.