but sometimes I feel bad when I have to tell him we already have plans
You need to stop feeling bad. He chose this situation and continues to choose it. It is his own doing. He knows how to pick up that phone and call his kids. It isn't your job to make it happen. He knows how to fix this situation with you. He is choosing not to. Stop feeling bad for how you think he might feel because of decisions he has chosen to make and continues to make.
Onward with your enjoyment and plans.
Kevin
Me 36, W 37 M: 08/02/97 D13, D9 1st Bomb 02/08 Reconciled 04/08 2nd Bomb: 09/08 W filed for D 02/04/09 Separated 03/09 D dismissed 06/09/09 Still separated...
