I have to agree with the others. Be yourself with the girls. You are a wonderful mother and don't let Andy's attitude affect that....ever!!! Just be yourself and ignore his idiotic ramblings. Say what you want, do what you want, etc. Stop walking on eggshells and letting his craziness affect you.
You will always be a part of the girl's lives. Andy will probably have many OW in the future that will come and go. You will always be their mother and should always be consistently in their lives. Make sure of that if at all possible.
I honestly see the time coming when Andy may not be emotionally able to care for the girls and that's one of the reasons you need to stay in contact with them. He seems to be spiralling out of control lately. Also, is there a law where you are that when they reach a certain age, they can choose which parent they want to live with?
I'd also look into "forcing" him to live up to the visitation agreement. If that means taking him to court, then do it.
Jo, no one here thinks you're a bad mom. I think I speak for everyone that we wish you had all the girls with you as you are by far the more stable parent.
You have to stop letting Andy's craziness get to you. The girls need a stable presence in their lives and you are it.