So ideally - you would have it written that he stays on the deed until you refi but you don't have to buy him out when you do refi, he just signs a quitclaim at that point, and IF you have to sell at a loss before then he pays half the loss. (The mortgage company will hold him to that anyway).

Can you get a roommate? An extra $300 or $400 a month would pay down your mortgage within 2 years instead of 4.