I could have written this post 5 years ago. I contacted OW's H as soon as I knew. Trouble was - he didn't know. She was even LIVING with my H and he didn't know. He didn't WANT to know. He was in denial.
I got the same response from H, although I hadn't told him. This man whom I had been with for 30 years shared all our phone conversations with her. SHE came on the line telling me what was mine and what they wanted in terms of MY material things. Can you imagine? That was the one and only time I ever spoke to that woman.
Many people have asked whether to tell OP's spouse or not. I still am not sure there is a right or wrong answer. Every situation is different. Not sure if it affected the outcome at all.
But I do understand your disbelief at your ex's response. It took me a long long time to realize that he was not the same person anymore so I could not predict his behaviour.
Anyway, just wanted to say "Been there, done that". Hope you got some answers. I DO feel that in the long run it did help me to understand things more.