I don'tmean to sound super dumb or anything but

why don't you refinance the house in your name and you can figure out what you need to pay him afterward?

why would he pay for anything to do with your refinancing if you are keeping the house

(I don't understand finance stuff)

if you refinance the house and it is in your name then you don't have to let him come in to your home to see the kids

if his name is still on it then it legally is still his, right

I would think that even if you lose the house and whatnot...I would sell the house just to get rid of the stress

it is a material thing
material things don't matter

peace of mind matters