I am no Vet, but if I was in your shoes, I would not have answered the phone. I would have let him wonder what you are up to. I think he knows you will pick up the phone and listen to him. Let him wonder. It could also be a boundary issue.
" I will not discuss things with you while you are with OW"
I'm sure one of the vets here can give you more advice and will correct me if I am wrong.
Me:44 EXW 44 Wonderful Children M11, T14 BD 6/14 OM Confirmed Divorce Final 2/25/16 "It works if you work it!"