Do you have an attorney yet? That would be an ideal question for them. What is going to happen to your house, the one you are in?
I tackled this once and simply told xH that "This is my place of peace and rest. You have yours (his dads) and I wouldn't do anything over there to make you uncomfortable. I know this is still 'your house', but please remember this is all the space I have".