"What I am doing is making sure my kids spend time with their dad."
It's not your job to keep everyone happy. Does he want to spend time with them? If so, then fine, let him make the arrangements. There's a fine line between being amicable and being a doormat.
Let me put it to you this way, If he was seeing someone else and you knew the person would be at his house, would you still happily take the kids to his place? Would you happily pass your kids along to let OW play with them?
I am not trying to make everyone happy. I care about my kids being happy, and right now, he has no means to get them, and considering he is being more than generous financially, it's the least I can do.
If he had an OW, yes, I would still take my kids. As I said, I can't control what H does, but if he chooses to find an OW, as long as she is good to my kids, I'd be happy. They come first. Always.