You are farther along in this so sometimes I don't feel like I'm qualified to give you advice... but I don't think it should be acceptable for him to live there AND come and go at unreasonable hours, AND I don't think you should ask where he's been. The solution? He needs to live somewhere else- if only for the sake of your kids. Will he listen if you tell him this is affecting them? Or will he ignore that too? Have you tried writing him a letter a la DB book (different medium) where you tell him this is unacceptable and he has x days to find a place to stay before you change the locks?
He's being a selfish dad and H - it's just disgusting when they can't even consider how their actions are affecting the kids, IMO. Wish I had more useful things to say. I hate how he's treating you guys.
(((Luv)))
When the men on the chessboard Get up and tell you where to go; And you've just had some kind of mushroom And your mind is moving slow; Go ask Alice... I think she'll know.