I am afraid you giving him a phone message may have done a bit of damage to your strong position on not wanting to talk to him while he's having an affair... its a mixed message is all...

YOu can repair this easily by sending him ONE more message saying :

"I am NOT taking anymore of your messages.. all calls are being redirected to your number.. I am NOT your doormat OR your secretary... take your own calls and leave me alone"

THAT will fix that mistake and you are back in business