I would also find a GOOD sitter and start going out at night. Go out to movies, restaurants, dress up... talk on the phone while he's there... act like he doens't matter... he's just a darn ghost.
If he challenges you and you HAVE to say something
"I am ready to work on this marriage once she's gone... Don't expect your family to wait for you forever..."