I would do what contact you must do with the chidlren, but you should not be involved in him contacting them... have someone else call the number and exit the call when done.
I would also make sure he knows he is NOT welcome back there to visit or on a permanent basis until the affair is ENDED
What you have there is good so far, but you arne't setting boundaries.. telling him teh affair must be ENDED is a sample boundary... otherwise you are just telling him you are his doormat and he can keep it up as long as he likes
Youc an at LEAST tell him YOU don't want to hear from him and the contact with his children will be minimal until the affair is over.