Julie, It's time to bit the bullet and get a visitation plan in place legally. I'm surprised that the school administrator did not escort the woman out of the building as she was not a parent of a student.
Your h is no longer a friend or someone you thought you knew. He's going to continue to push your buttons and the best way to do that is thru your child for now. Protect your child and yourself...call a lawyer today. Don't put this off another day.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.