Awkward - sure. Tough to do - sure. This could be the best time to keep contact now that you have just had contact with his family. Keep that contact going so that H can't play the game between everyone.
The kids would like to go, make it a happy memorable holiday.
When my H's kids were younger some stayed with us and sometimes stayed with their mom. We all spent many holidays together, graduation, birthdays, weddings, etc. It's awkward if you let it be that way, it's not if you don't. I remember H's xw, xmil, xsil all came to my house and did all the cooking in my kitchen for oldest son's graduation party. Worked for me! I didn't have to worry about them talking about my cooking!
Anyway, that's just my 2 cents.
Live your life while you are still living. Riding the trail less traveled.
