I had a live in for about 15 months post bomb. Like Cat said, find a sanctuary. Looking back, I would say that having a live in was much tougher for me.
The focus has to be on you and the kids. That doesn't mean you have to be rude or ignore him, but finding some kind of middle ground where you can talk like friends/housemates (whatever you want to call it) is hard.
With all that you've been through, it's easy to see that you have high tensile steel in your veins. Doesn't mean it won't be hard, but it does mean you have alot more power than you think.