You should at least respond that you are NOT poisoning the kids against him. Just to have documentation should it be needed in the future. These guys cannot face the fact that it is THEIR actions that are causing their kids' responses to them. But there is a whole "parental alienation" issue out there that the courts buy into, so protect yourself by calmly disputing his accusations.
Also - use some emotional Aikido in the process, such as "Ex, I know you only want what is best for the kids and me. I am trying my best to encourage their relationship with you and I never badmouth you to them. Unfortunately, they are still aware of the circumstances of your affair and concerned about my life-threatening illness, so you have a lot to overcome with them. I think your best approach would be to spend time alone with them, so that they feel that they are your priority. Don't try so hard to rush them into a relationship with OW - that will come slowly. Perhaps you might consider going to counseling with the kids?"
