Ahem.....

No way in hell does he get to run you out of your own house just because he would be uncomfortable with you there.

You cannot trust a man who walks out on you. You have every right to be there when he comes over.