It's not your responsibility to deal with his choices. Of course, if he can brow beat you into doing it he feels better. You are making the right decisions here. Don't let him pull you in.
If he wants to see his d he will make the effort. He is an ass. Stay away from it and continue to make a stable place for you and your D's.