he needs to get out of your house. I know you hope he will come around but I don't think this situation is going to help and it seems it is prolonging a weird status. He is not going to be missing "the home" you two had if he still comes in and plays house with your Ds with or without you.