In my opinion, H knows he is hurting me (him hearing me cry yesterday morning as he was leaving) and he doesn't know how to deal with it. The answer is simple. Stop the affair and come home. He does not want to do that. He has moved to another state that I guess he likes, he has OW with him, and he probably thinks he is in too deep to get out.
You have the idea for avoiding incidental contact, he uses the contact to validate his affair. As long as you accept his "I'm back" and his infrequent visits that celebrate his return he becomes increasingly convinced everyone is in support of his lifestyle and indirectly this is supporting the damage done to you and your chidlren. When he is supported, his mistreatment of YOU is supported with it.
He can't support his lifestyle long term. I would get yourself in a great position while he's gone. His distance offers the benefit of you being very far apart from his drama. Some posters here have the infidelity and the drama in there very own home and in many cases thats a lot more painful.