I think you should do what you and your kids feel most comfortable with. My D's definitely want to be here, in the house they refer to as home. But I'm not inviting H. If you are comfortable inviting him, you shouldn't have to form your plans around where he will be 'most comfortable.' It's your Christmas, too.
