I don't feel angst about saying "no" but there is constant blowback.
I constantly hear how the kids are "struggling" because of what I did or didn't do. She makes it sound like the children are scarred for life because, for instance, I had to move houses recently (I rent). Or I went on a hiking trip and they were worried about my safety. Or I enrolled them in some after-school care.
I don't like the games going on with my kids, but I don't think there's much I can do other than let it go and not react.