Jo, What is your "legal" visition for you girls? Is this something that can be changed to your advantage and their benefit. No matter what happens between you and Andy, it is "not" right for him to hold the girls hostage to control you. He is extremely selfish in keeping them away from you and not a good roll model for your children.
Why not look into legal ways to spend more time and have a larger roll in your girls life? After that is accomplished, if you still want Andy, work on the DBing. In the meantime, it gives you a mission to focus on and if it means stepping on his toes for awhile, so be it. You have nothing to lose, it seems.