I don't know your complete sitch, but from what you wrote above and your signature line - "Came back...too many times!!" - I think you need to consider a couple of things. First what is this doing to you and your kids with him being so indecisive coming and going? There is no stability in that and only heartache for you and your children continuously dealing with their father checking in and out of their "home". Secondly, ask yourself what's different this time compared to the other times he's thought about coming back. If it's the same old song and dance then I'd take a pass for now. If it seems different, more sincere, then I think for you to even consider trying this again, you need to sit down with your H and set out clear ground rules and boundaries which include him not seeing other women obviously, especially those who are not a "friend" of the marriage. Just my 2 cents. I wish you the best.
