The ow my H is with also resents the time he spends with the kids. I think it's the insecurity they must have when they've taken up with someone who's M with children.
I hope your H and S have a wonderful time and it's possible it may remind H of what he has been missing out on.