Print out that last paragraph and read it every morning. Remember it when he decides its time to play happy family again.
And once you see him and know he isn't high, let a family member sit and babysit with him - go out, get as far away as you can from this guy....ugh.
However, I'd pursue the option of getting an officer of Family Court, Child Protective Services or some such agency to do some of the supervising - professionally - for you. You need the break. And, it's like you are supervising two children!
Gardener
"My soul, be satisfied with flowers, With fruit, with weeds even; but gather them In the one garden you may call your own." Cyrano deBergerac